Giambattista Bloisi created SPARK-44634:
-------------------------------------------
Summary: Encoders.bean does no longer support nested beans with
type arguments
Key: SPARK-44634
URL: https://issues.apache.org/jira/browse/SPARK-44634
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.4.1, 3.5.0, 4.0.0
Reporter: Giambattista Bloisi
Hi,
while upgrading a project from spark 2.4.0 to 3.4.1 version, I have
encountered the same problem described in [java - Encoders.bean attempts to
check the validity of a return type considering its generic type and not its
concrete class, with Spark 3.4.0 - Stack
Overflow|https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge].
Put it short, starting from Spark 3.4.x Encoders.bean throws an exception when
the passed class contains a field whose type is a nested bean with type
arguments:
{code:java}
class A<T> {
T value;
// value getter and setter
}
class B {
A<String> stringHolder;
// stringHolder getter and setter
}
Encoders.bean(B.class); // throws "SparkUnsupportedOperationException:
[ENCODER_NOT_FOUND]..."{code}
It looks like this is a regression introduced with [[SPARK-42093][SQL] Move
JavaTypeInference to AgnosticEncoders - apache/spark@1867200 (github.com)
|https://github.com/apache/spark/commit/18672003513d5a4aa610b6b94dbbc15c33185d3a#diff-1191737b908340a2f4c22b71b1c40ebaa0da9d8b40c958089c346a3bda26943b]while
getting rid of TypeToken, that somehow managed that case.
I'm going to submit a PR to re-enable this functionality.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
