[
https://issues.apache.org/jira/browse/SPARK-53790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-53790:
-----------------------------------
Labels: pull-request-available (was: )
> arrow encoder is unable to handle Beans with instantiated generics
> ------------------------------------------------------------------
>
> Key: SPARK-53790
> URL: https://issues.apache.org/jira/browse/SPARK-53790
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.1.0
> Reporter: Steven Aerts
> Priority: Major
> Labels: pull-request-available
>
> We found a bug in the arrow encoder/decoder for spark connect which is unable
> to handle beans with instantiated generics. Like:
> ```
> class JavaBeanWithGenerics[T] {
> @BeanProperty var value: T = _
> }
> class JavaBeanWithGenericsWrapper {
> @BeanProperty var value: JavaBeanWithGenerics[String] = _
> }
> ...
> val encoder =
> JavaTypeInference.encoderFor(classOf[JavaBeanWithGenericsWrapper])
> ```
> Which results in the following error when the above encoder is used in
> connect:
> ```
> java.lang.NoSuchMethodException: no such method:
> JavaBeanWithGenerics.getValue()String/invokeVirtual
> at
> java.base/java.lang.invoke.MemberName.makeAccessException(MemberName.java:915)
> at
> java.base/java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:994)
> at
> java.base/java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:3750)
> at
> java.base/java.lang.invoke.MethodHandles$Lookup.findVirtual(MethodHandles.java:2767)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.$anonfun$serializerFor$19(ArrowSerializer.scala:495)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.$anonfun$serializerFor$19$adapted(ArrowSerializer.scala:491)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.$anonfun$structSerializerFor$1(ArrowSerializer.scala:549)
> at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:75)
> at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:35)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.structSerializerFor(ArrowSerializer.scala:547)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.serializerFor(ArrowSerializer.scala:491)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.$anonfun$structSerializerFor$1(ArrowSerializer.scala:548)
> at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:75)
> at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:35)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.structSerializerFor(ArrowSerializer.scala:547)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.serializerFor(ArrowSerializer.scala:491)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.serializerFor(ArrowSerializer.scala:237)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer.<init>(ArrowSerializer.scala:62)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$$anon$1.<init>(ArrowSerializer.scala:161)
> at
> org.apache.spark.sql.connect.client.arrow.ArrowSerializer$.serialize(ArrowSerializer.scala:160)
> ```
> We have have a possible patch for this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
