maropu commented on a change in pull request #31888:
URL: https://github.com/apache/spark/pull/31888#discussion_r599324821
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
##########
@@ -203,6 +203,36 @@ class ExpressionEncoderSuite extends
CodegenInterpretedPlanTest with AnalysisTes
encodeDecodeTest(Array(Option(InnerClass(1))), "array of optional inner
class")
+ // NOTE: branch-2.4 does not have the interpreted implementation of
SafeProjection, so
+ // it does not fall back into the interpreted mode if the compilation fails.
+ // Therefore, the test in this PR just checks that the compilation error
happens
+ // instead of checking that the interpreted mode works well.
+ private def checkCompilationError[T : ExpressionEncoder](
+ input: T,
+ testName: String): Unit = {
+ testAndVerifyNotLeakingReflectionObjects(s"compilation error: $testName:
$input") {
+ val encoder = implicitly[ExpressionEncoder[T]]
+ val row = encoder.toRow(input)
+ val boundEncoder = encoder.resolveAndBind()
+ val errMsg = intercept[RuntimeException] {
+ boundEncoder.fromRow(row)
+ }.getCause.getMessage
+ assert(errMsg.contains("failed to compile: "))
Review comment:
> Is this the compilation error?
Yea, that's the expected error. Actually, the master has same error when
`useFallback = false`:
https://github.com/apache/spark/blob/d32bb4e5ee4718741252c46c50a40810b722f12d/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala#L219
```
// master
[info] Cause: org.codehaus.commons.compiler.CompileException: File
'generated.java', Line 32, Column 124: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32,
Column 124:
"org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$MalformedClassObject$"
has no member type
"ExpressionEncoderSuite$MalformedClassObject$MalformedNameExample"
[info] at
org.apache.spark.sql.errors.QueryExecutionErrors$.compilerError(QueryExecutionErrors.scala:328)
```
> But by moving MalformedClassObject out of ExpressionEncoderSuite, the
generated code can be compiled.
IIUC the condition where the error happens is that `MalformedClassObject` is
placed in `ExpressionEncoderSuite` as Kris said in the previous PR:
https://github.com/apache/spark/pull/31709#issuecomment-791275810
```
Actually, what's more interesting is the different rules of Java's nested
classes and Scala's. Since there is no such
notion of a "companion object" or built-in "singleton object" notion on the
Java language level, at least not in
Java 8, the Java language syntax for creating a new instance of a inner
class object doesn't work well for Scala's
class Foo { object Bar { class Baz {...} } } kind of nesting, i.e. even
when we do use the correct "simple name"
(e.g. MalformedNameExample), there would still be no valid Java syntax to
create it (i.e. outerObj.new
InnerClass(...) doesn't work when outerObj is a Scala object)
```
cc: @rednaxelafx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
