[
https://issues.apache.org/jira/browse/SPARK-19742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887648#comment-15887648
]
Navin Goel commented on SPARK-19742:
------------------------------------
I would disagree on this approach, as the Dataset already has the schema
specified which is matching the Hive schema I would expect it to insert in the
proper columns. Otherwise it would be impossible to insert into an already
existing table to the proper columns specially if you are not inserting value
into every column.
So at this point my option is to create a new table each time?
I am using spark streaming job and updating a table in Hive from the data that
I receive, creating a new table for each table is just not feasible... is
there a workaround this issue?
> When using SparkSession to write a dataset to Hive the schema is ignored
> ------------------------------------------------------------------------
>
> Key: SPARK-19742
> URL: https://issues.apache.org/jira/browse/SPARK-19742
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.0.1
> Environment: Running on Ubuntu with HDP 2.4.
> Reporter: Navin Goel
>
> I am saving a Dataset that is created form reading a json and some selects
> and filters into a hive table. The dataset.write().insertInto function does
> not look at schema when writing to the table but instead writes in order to
> the hive table.
> The schemas for both the tables are same.
> schema printed from spark of the dataset being written:
> StructType(StructField(countrycode,StringType,true),
> StructField(systemflag,StringType,true),
> StructField(classcode,StringType,true),
> StructField(classname,StringType,true),
> StructField(rangestart,StringType,true),
> StructField(rangeend,StringType,true),
> StructField(tablename,StringType,true),
> StructField(last_updated_date,TimestampType,true))
> Schema of the dataset after loading the same table from Hive:
> StructType(StructField(systemflag,StringType,true),
> StructField(RangeEnd,StringType,true),
> StructField(classcode,StringType,true),
> StructField(classname,StringType,true),
> StructField(last_updated_date,TimestampType,true),
> StructField(countrycode,StringType,true),
> StructField(rangestart,StringType,true),
> StructField(tablename,StringType,true))
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
