[jira] [Commented] (SPARK-19580) Support for avro.schema.url while writing to hive table

2017-11-22 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262484#comment-16262484
 ] 

Apache Spark commented on SPARK-19580:
--

User 'vinodkc' has created a pull request for this issue:
https://github.com/apache/spark/pull/19795

> Support for avro.schema.url while writing to hive table
> ---
>
> Key: SPARK-19580
> URL: https://issues.apache.org/jira/browse/SPARK-19580
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.6.3, 2.1.0
>Reporter: Mateusz Boryn
>Priority: Critical
>
> Support for writing to Hive table which uses Avro schema pointed to by 
> avro.schema.url is missing. 
> I have Hive table with Avro data format. Table is created with query like 
> this:
> {code:sql}
> CREATE TABLE some_table
>   PARTITIONED BY (YEAR int, MONTH int, DAY int)
>   ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION 'hdfs:///user/some_user/some_table'
>   TBLPROPERTIES (
> 'avro.schema.url'='hdfs:///user/some_user/some_table.avsc'
>   )
> {code}
> Please notice that there is `avro.schema.url` and not `avro.schema.literal` 
> property, as we have to keep schemas in separate files for some reasons.
> Trying to write to such table results in NPE.
> Tried to find workaround for this, but nothing helps. Tried:
> - setting df.write.option("avroSchema", avroSchema) with explicit schema 
> in string
> - changing TBLPROPERTIES to SERDEPROPERTIES
> - replacing explicit detailed SERDE specification with STORED AS AVRO
> I found that this can be solved by adding a couple of lines in 
> `org.apache.spark.sql.hive.HiveShim` next to 
> `AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL` is referenced.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19580) Support for avro.schema.url while writing to hive table

2017-11-18 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257985#comment-16257985
 ] 

Apache Spark commented on SPARK-19580:
--

User 'vinodkc' has created a pull request for this issue:
https://github.com/apache/spark/pull/19779

> Support for avro.schema.url while writing to hive table
> ---
>
> Key: SPARK-19580
> URL: https://issues.apache.org/jira/browse/SPARK-19580
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.6.3, 2.1.0
>Reporter: Mateusz Boryn
>Priority: Critical
>
> Support for writing to Hive table which uses Avro schema pointed to by 
> avro.schema.url is missing. 
> I have Hive table with Avro data format. Table is created with query like 
> this:
> {code:sql}
> CREATE TABLE some_table
>   PARTITIONED BY (YEAR int, MONTH int, DAY int)
>   ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION 'hdfs:///user/some_user/some_table'
>   TBLPROPERTIES (
> 'avro.schema.url'='hdfs:///user/some_user/some_table.avsc'
>   )
> {code}
> Please notice that there is `avro.schema.url` and not `avro.schema.literal` 
> property, as we have to keep schemas in separate files for some reasons.
> Trying to write to such table results in NPE.
> Tried to find workaround for this, but nothing helps. Tried:
> - setting df.write.option("avroSchema", avroSchema) with explicit schema 
> in string
> - changing TBLPROPERTIES to SERDEPROPERTIES
> - replacing explicit detailed SERDE specification with STORED AS AVRO
> I found that this can be solved by adding a couple of lines in 
> `org.apache.spark.sql.hive.HiveShim` next to 
> `AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL` is referenced.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19580) Support for avro.schema.url while writing to hive table

2017-02-20 Thread Mateusz Boryn (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-19580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874785#comment-15874785
 ] 

Mateusz Boryn commented on SPARK-19580:
---

Similar to SPARK-17920

> Support for avro.schema.url while writing to hive table
> ---
>
> Key: SPARK-19580
> URL: https://issues.apache.org/jira/browse/SPARK-19580
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.6.3, 2.1.0
>Reporter: Mateusz Boryn
>Priority: Critical
>
> Support for writing to Hive table which uses Avro schema pointed to by 
> avro.schema.url is missing. 
> I have Hive table with Avro data format. Table is created with query like 
> this:
> {code:sql}
> CREATE TABLE some_table
>   PARTITIONED BY (YEAR int, MONTH int, DAY int)
>   ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION 'hdfs:///user/some_user/some_table'
>   TBLPROPERTIES (
> 'avro.schema.url'='hdfs:///user/some_user/some_table.avsc'
>   )
> {code}
> Please notice that there is `avro.schema.url` and not `avro.schema.literal` 
> property, as we have to keep schemas in separate files for some reasons.
> Trying to write to such table results in NPE.
> Tried to find workaround for this, but nothing helps. Tried:
> - setting df.write.option("avroSchema", avroSchema) with explicit schema 
> in string
> - changing TBLPROPERTIES to SERDEPROPERTIES
> - replacing explicit detailed SERDE specification with STORED AS AVRO
> I found that this can be solved by adding a couple of lines in 
> `org.apache.spark.sql.hive.HiveShim` next to 
> `AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL` is referenced.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org