GitHub user fangshil opened a pull request:
https://github.com/apache/spark/pull/20974
[SPARK-23862][SQL] Spark ExpressionEncoder should support java enum type in
scala
## What changes were proposed in this pull request?
In SPARK-21255, spark upstream adds support for creating encoders for java
enum types, but the support is only added to Java API(for enum working within
Java Beans). Since the java enum can come from third-party java library, we
have use case that requires
1. using java enum types as field of scala case class
2. using java enum as the type T in Dataset[T]
Spark ExpressionEncoder already supports ser/de many java types in
ScalaReflection, so we propose to add support for java enum as well, as a
follow up of SPARK-21255.
## How was this patch tested?
Tested the patch in our production cluster. Added unit test.
Since:
1. it is not possible to define a java enum in scala directly, since the
defined enum class in scala will miss method like valueOf which is added by
java compiler
2. it is not possible to define a test enum java class and use in scala
test because the compilation of single scala
test(-DwildcardSuites=org.apache.spark.sql.DatasetSuite) won't compile the test
java class first
As a result, I use the Spark SQL public java enum API(SaveMode.java) in the
test. Please advise if there is a better way to test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fangshil/spark SPARK-23862
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20974.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20974
----
commit 90effb21375a2ec0e93426efcaae092ad3f59e26
Author: Fangshi Li <fli@...>
Date: 2018-04-04T04:52:36Z
SPARK-23862: Spark ExpressionEncoder should support java enum type in scala
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
