Hi Team, I am currently working on running Spark SQL core unit tests using spark-submit(3.5.4) on a YARN cluster and have encountered several challenges that I hope you can help me address.Background: I am attempting to execute the Spark SQL core unit tests packaged within a JAR file (spark-sql_2.12-3.5.4-SNAPSHOT-tests.jar) using ScalaTest's org.scalatest.tools.Runner as the main class. The goal is to run these tests directly on a YARN cluster to validate our Spark SQL functionalities in a distributed environment.Issue Encountered:Despite configuring the spark-submit command with appropriate class paths and JAR files, the application fails to locate the org.scalatest.tools.Runner class, resulting in a ClassNotFoundException. Here is the spark-submit command I've been using:
/bin/spark-submit \ --master yarn \ --deploy-mode cluster \ --class org.scalatest.tools.Runner \ --jars /opt/spark/scalatest_2.12-3.2.10.jar,/opt/spark/scala-xml_3-2.1.0.jar \ --driver-class-path /opt/spark/scalatest_2.12-3.2.10.jar:/opt/spark/scala-xml_3-2.1.0.jar \ /opt/spark/spark-sql_2.12-3.5.4-SNAPSHOT-tests.jar \ -o -s org.apache.spark.sql.SQLQueryTestSuite Any insights or recommendations on how to properly configure the environment or spark-submit command to successfully execute these tests? Thank you for your time and assistance. Best Regards, Satya Kommula.
