GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/21472
[SPARK-24445][SQL] Schema in json format for from_json in SQL
## What changes were proposed in this pull request?
In the PR, I propose to support schema in JSON format for the `from_json()`
function in SQL as it has been already implemented in Scala DSL for example
[there](https://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L3225-L3229):
```scala
val dataType = try {
DataType.fromJson(schema)
} catch {
case NonFatal(_) => StructType.fromDDL(schema)
}
```
The changes will allow to specify `MapType` in SQL that's impossible at the
moment:
```sql
select from_json('{"a":1}', '{"type":"map", "keyType":"string",
"valueType":"integer","valueContainsNull":false}')
```
## How was this patch tested?
Added a couple cases to `json-functions.sql`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 from_json-sql-schema
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21472.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21472
----
commit 778162074cce7046f5f48d02780c8bac251fe96b
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-05-31T18:49:23Z
SQL tests for schema in json format
commit 139ef7e98c27c5f24d7a25fec8789309657cb5d3
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-05-31T19:42:27Z
Support schema in json format
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
