GitHub user yijieshen reopened a pull request:
https://github.com/apache/spark/pull/6874
[SPARK-8407][SQL]complex type constructors: struct and named_struct
This is a follow up of
[SPARK-8283](https://issues.apache.org/jira/browse/SPARK-8283)
([PR-6828](https://github.com/apache/spark/pull/6828)), to support both
`struct` and `named_struct` in Spark SQL.
After [#6725](https://github.com/apache/spark/pull/6828), the semantic of
[`CreateStruct`](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypes.scala#L56)
methods have changed a little and do not limited to cols of
`NamedExpressions`, it will name non-NamedExpression fields following the hive
convention, col1, col2 ...
This PR would both loosen
[`struct`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L723)
to take children of `Expression` type and add `named_struct` support.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yijieshen/spark SPARK-8283
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6874.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6874
----
commit 677e0b7fda584ce09b042642a68d98ce530fdfb9
Author: Yijie Shen <[email protected]>
Date: 2015-06-15T08:33:02Z
Resolve udf_struct test failure by automatically generate structField name
for non-NamedExpression children
commit 6052b737d7a252a522fdbf290a8ce502b7082a54
Author: Yijie Shen <[email protected]>
Date: 2015-06-16T09:50:27Z
Doc fix
commit ecff7836c1a7309b61d31ef657d2702f5ca6213b
Author: Yijie Shen <[email protected]>
Date: 2015-06-18T07:16:08Z
Add CreateNamedStruct in both DataFrame function API and FunctionRegistery
commit 62facf293f5c6020f6844ef665d774eee7c9bad1
Author: Yijie Shen <[email protected]>
Date: 2015-06-18T08:41:32Z
Merge remote-tracking branch 'apache-spark/master' into SPARK-8283
commit 869b943b3a523aeacc61107e0ad15159db7bd4dc
Author: Yijie Shen <[email protected]>
Date: 2015-06-18T09:13:00Z
loosen struct method in functions.scala to take Expression children
commit 64e4bf78a5a249eae5e896da0b0b1d6e4a8a5d1c
Author: Yijie Shen <[email protected]>
Date: 2015-06-19T04:34:33Z
Fix reviews
commit 61083ab7e5128db433004fe5c996077e7697c826
Author: Yijie Shen <[email protected]>
Date: 2015-06-19T05:56:14Z
remove nameStruct API from DataFrame
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
