GitHub user yijieshen opened a pull request:
https://github.com/apache/spark/pull/6874
[SPARK-8407]complex type constructors: struct and named_struct
This is a follow up of
[SPARK-8283](https://issues.apache.org/jira/browse/SPARK-8283)
([PR-6828](https://github.com/apache/spark/pull/6828)), to support both
`struct` and `named_struct` in Spark SQL.
After [PR-6828](https://github.com/apache/spark/pull/6828), the semantic of
[`CreateStruct`](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypes.scala#L56)
methods have changed a little and do not limited to cols of
`NamedExpressions`, it will name non-NamedExpression fields following the hive
convention, col1, col2 ...
This PR would both loosen
[`struct`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L723)
to take children of `Expression` type and add `named_struct` support.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yijieshen/spark SPARK-8283
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6874.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6874
----
commit 677e0b7fda584ce09b042642a68d98ce530fdfb9
Author: Yijie Shen <[email protected]>
Date: 2015-06-15T08:33:02Z
Resolve udf_struct test failure by automatically generate structField name
for non-NamedExpression children
commit 6052b737d7a252a522fdbf290a8ce502b7082a54
Author: Yijie Shen <[email protected]>
Date: 2015-06-16T09:50:27Z
Doc fix
commit ecff7836c1a7309b61d31ef657d2702f5ca6213b
Author: Yijie Shen <[email protected]>
Date: 2015-06-18T07:16:08Z
Add CreateNamedStruct in both DataFrame function API and FunctionRegistery
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
