GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/13860
[SPARK-16157] [SQL] Add New Methods for comments in StructField and
StructType
#### What changes were proposed in this pull request?
Based on the previous discussion with @cloud-fan @hvanhovell in another
related PR https://github.com/apache/spark/pull/13764#discussion_r67994276, it
looks reasonable to add convenience methods for users to add `comment` when
defining `StructField`.
Currently, the column-related `comment` attribute is stored in `Metadata`
of `StructField`. For example,
```Scala
StructType(
StructField(
"cl1",
IntegerType,
nullable = false,
new MetadataBuilder().putString("comment", "test").build()) :: Nil)
```
This PR is to add user friendly methods for the `comment` attribute when
defining a `StructField`. After the changes, users are provided three different
ways to do it:
```Scala
val struct = (new StructType)
.add("a", "int", true, "test1")
val struct = (new StructType)
.add(StructField("b", LongType, false, "test2"))
val struct = (new StructType)
.add("c", StringType, true, "test3")
```
#### How was this patch tested?
Added test cases:
- `DataTypeSuite` is for testing three types of API changes,
- `DataFrameReaderWriterSuite` is for parquet, json and csv formats - using
in-memory catalog
- `OrcQuerySuite.scala` is for orc format using Hive-metastore
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark newMethodForComment
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13860.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13860
----
commit 2826e28e10fcec71d3ab1dab3f78dc97f3535b3b
Author: gatorsmile <[email protected]>
Date: 2016-06-22T20:34:14Z
add new APIs
commit a0c3f9ff3d588b0c8b6cb9723c6d4bb2dcd78366
Author: gatorsmile <[email protected]>
Date: 2016-06-22T20:35:16Z
Merge remote-tracking branch 'upstream/master' into newMethodForComment
commit 06667c0fc2da54ae2167b41e35bd9f438d6eeb65
Author: gatorsmile <[email protected]>
Date: 2016-06-22T21:49:43Z
add test case
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
