GitHub user jinxing64 opened a pull request:
https://github.com/apache/spark/pull/19652
[SPARK-22435][SQL] Support processing array and map type using script
## What changes were proposed in this pull request?
Currently, It is not supported to use script(e.g. python) to process array
type or map type, it will complain with below message:
`org.apache.spark.sql.catalyst.expressions.UnsafeArrayData cannot be cast
to [Ljava.lang.Object`
`org.apache.spark.sql.catalyst.expressions.UnsafeMapData cannot be cast to
java.util.Map`
This pr proposes to support it by using `DelimitedJSONSerDe`
This pr also fixes a bug -- when using input row format with script, no
data will be produced from `ScriptTransformationExec`.
## How was this patch tested?
Tests added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jinxing64/spark SPARK-22435
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19652.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19652
----
commit 4a994260e5be1ea8ee204f70b0f025bbab76e542
Author: jinxing <[email protected]>
Date: 2017-11-03T09:39:48Z
Support processing array and map type using script
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
