[
https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961783#comment-16961783
]
Aman Omer edited comment on SPARK-29595 at 10/29/19 8:17 AM:
-------------------------------------------------------------
{color:#172b4d}Parameter required for named_struct is Seq(name1, val1, name2,
val2, ...). Validation step for named_struct only check for string at odd
places. For example following query will add a row in _str_ table.{color}
{code:java}
insert into str values named_struct( "ab", 1, "ba", 2);{code}
According to the discussion in [https://github.com/apache/spark/pull/26275] ,
which was tackling similar issue, changing fields of struct type according to
names will introduce complexity.
So I think Spark should throw an exception when names does not match in
named_struct.
cc [~srowen] [~maropu]
was (Author: aman_omer):
{color:#172b4d}Parameters required for named_struct is Seq(name1, val1, name2,
val2, ...). Validation step for named_struct only check for string at odd
places. For example following query will add a row in _str_ table.{color}{color}
{code:java}
insert into str values named_struct( "ab", 1, "ba", 2);{code}
According to the discussion in [https://github.com/apache/spark/pull/26275] ,
which was tackling similar issue, changing fields of struct type according to
names will introduce complexity.
So I think Spark should throw an exception when names does not match in
named_struct.
cc [~srowen] [~maropu]
> Insertion with named_struct should match by name
> ------------------------------------------------
>
> Key: SPARK-29595
> URL: https://issues.apache.org/jira/browse/SPARK-29595
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Gengliang Wang
> Priority: Major
>
> {code:java}
> spark-sql> create table str using parquet as(select named_struct('a', 1, 'b',
> 2) as data);
> spark-sql> insert into str values named_struct("b", 3, "a", 1);
> spark-sql> select * from str;
> {"a":3,"b":1}
> {"a":1,"b":2}
> {code}
> The result should be
> {code:java}
> {"a":1,"b":3}
> {"a":1,"b":2}
> {code}
> Spark should match the field names of named_struct on insertion
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
