[
https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andoni Teso updated SPARK-46679:
--------------------------------
Priority: Blocker (was: Major)
> Encoders with multiple inheritance - Key not found: T
> -----------------------------------------------------
>
> Key: SPARK-46679
> URL: https://issues.apache.org/jira/browse/SPARK-46679
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.2, 3.5.0
> Reporter: Andoni Teso
> Priority: Blocker
> Attachments: spark_test.zip
>
>
> Since version 3.4, I've been experiencing the following error when using
> encoders.
> {code:java}
> Exception in thread "main" java.util.NoSuchElementException: key not found: T
> at scala.collection.immutable.Map$Map1.apply(Map.scala:163)
> at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:121)
> at
> org.apache.spark.sql.catalyst.JavaTypeInference$.$anonfun$encoderFor$1(JavaTypeInference.scala:140)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at scala.collection.TraversableLike.map(TraversableLike.scala:286)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
> at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:138)
> at
> org.apache.spark.sql.catalyst.JavaTypeInference$.$anonfun$encoderFor$1(JavaTypeInference.scala:140)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at scala.collection.TraversableLike.map(TraversableLike.scala:286)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
> at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:138)
> at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:60)
> at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:53)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.javaBean(ExpressionEncoder.scala:62)
> at org.apache.spark.sql.Encoders$.bean(Encoders.scala:179)
> at org.apache.spark.sql.Encoders.bean(Encoders.scala)
> at org.example.Main.main(Main.java:26) {code}
> I'm attaching the code I use to reproduce the error locally.
> [^spark_test.zip]
> The issue is in the JavaTypeInference class when it tries to find the encoder
> for a ParameterizedType with the value Team<T>. When running
> JavaTypeUtils.getTypeArguments(pt).asScala.toMap, it returns the type T
> again, but this time as a Company object, and pt.getRawType as Team. This
> ends up generating a tuple of Team, Company in the typeVariables map, leading
> to errors when searching for TypeVariables.
> In my case, I've resolved this by doing the following:
> {code:java}
> case tv: TypeVariable[_] =>
> encoderFor(typeVariables.head._2, seenTypeSet, typeVariables)
> case pt: ParameterizedType =>
> encoderFor(pt.getRawType, seenTypeSet, typeVariables) {code}
> I haven't submitted a pull request because it doesn't seem to be the most
> optimal solution, or it might break some parts of the code. Additional
> validations or conditions may need to be added.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
