maropu commented on pull request #31747: URL: https://github.com/apache/spark/pull/31747#issuecomment-799938197
I noticed that `branch-2.4` does not have an interpreted implementation of `SafeProjection`, so the fallback cannot happen in some code path. For example, `ExpressionEncoder` directly uses `GenerateSafeProjection` (see the code below), so it throws an exception no matter what value we set to `spark.sql.codegen.factoryMode`. https://github.com/apache/spark/blob/3c627ad6f6392b085e05475a7475f7471190d0a8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala#L271 I think we have two options; - give up this PR and #31709 in branch-2.4 - merge this PR and #31709 into branch-2.4 for changing the thrown exception from JVM `StringIndexOutOfBoundsException` into a Janino compilation error I feel there is no much difference between them for users though, the second one looks better cuz `StringIndexOutOfBoundsException` gives users no error context, I think. WDHY? @viirya @dongjoon-hyun @HyukjinKwon ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
