GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/11752
[SPARK-3308][SQL][FOLLOW-UP] Parse JSON rows having an array type and a
struct type in the same fieild
## What changes were proposed in this pull request?
This https://github.com/apache/spark/pull/2400 added the support to parse
JSON rows wrapped with an array. However, when the given data contains array
data and struct data in the same field as below:
```json
{"a": {"b": 1}}
{"a": []}
```
and the schema is given as below:
```scala
val schema =
StructType(
StructField("a", StructType(
StructField("b", StringType) :: Nil
)) :: Nil)
```
This throws an exception below:
```java
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 7 in stage 0.0 failed 4 times, most recent failure: Lost
task 7.3 in stage 0.0 (TID 10, 192.168.1.170): java.lang.ClassCastException:
org.apache.spark.sql.types.GenericArrayData cannot be cast to
org.apache.spark.sql.catalyst.InternalRow
at
org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getStruct(rows.scala:50)
at
org.apache.spark.sql.catalyst.expressions.GenericMutableRow.getStruct(rows.scala:247)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.eval(Unknown
Source)
...
```
For other data types, in this case it converts the given values are `null`
but only this case emits an exception.
This PR makes the support for wrapped rows applied only at the top level.
## How was this patch tested?
Unit tests were used and `./dev/run_tests` for code style tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-3308-follow-up
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11752.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11752
----
commit 4ab1e80e8bca868fc2387cf74004cc44b6e4664a
Author: hyukjinkwon <[email protected]>
Date: 2016-03-16T04:36:56Z
Parse JSON rows having an array type and a struct type in the same field.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
