yew1eb commented on issue #1961: URL: https://github.com/apache/auron/issues/1961#issuecomment-3819257116
Spark 4.1 unit tests encounter Catalyst codegen failure due to incompatibility between its newly introduced and default-enabled `spark.sql.artifacts.sessionIsolation.enabled` and the original REPL class server logic: this isolation mechanism stores Catalyst dynamic inline classes like `org.apache.spark.sql.catalyst.expressions.Object` in unique session-specific directories for each SparkSession (worsened by `SharedSparkSession` creating independent sessions per test), but the REPL class server’s scanning path is still hardcoded to the global shared directory, and Spark 4.x’s mandatory REPL class pre-validation logic triggers a check failure when the server cannot locate the above class, blocking the codegen pipeline. This issue does not exist in Spark 3.x because it lacks the session isolation mechanism entirely (all dynamic classes are stored in the scannable global directory), and it only enables REPL for interactive scenarios while disabling it by default for unit tests wit h no mandatory REPL pre-validation, thus avoiding any path mismatch and subsequent failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
