LuciferYang opened a new pull request #29370: URL: https://github.com/apache/spark/pull/29370
### What changes were proposed in this pull request? The purpose of this pr is to partial resolve [SPARK-32526](https://issues.apache.org/jira/browse/SPARK-32526), fix `DataSourceV2AnalysisBaseSuite` related test cases and `TreeNodeSuite`, 72 failed cases in `sql/catalyst` module. The main change of this pr as following: - `Optimizer` and `Analyzer` are changed to pass compile, `ArrayBuffer` is not a `Seq` in scala 2.13, call `toSeq` method manually to compatible with Scala 2.12 - `m.mapValues().view.force` pattern return a `Map` in scala 2.12 but return a `IndexedSeq` in scala 2.13, call `toMap` method manually to compatible with Scala 2.12. `TreeNode` are changed to pass `DataSourceV2AnalysisBaseSuite` related test cases and `TreeNodeSuite` failed case. ### Why are the changes needed? We need to support a Scala 2.13 build. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Scala 2.12: Pass the Jenkins or GitHub Action - Scala 2.13: Do the following: ``` $ dev/change-scala-version.sh 2.13 ``` **Before** ``` mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2AppendDataANSIAnalysisSuite -Dtest=none Tests: succeeded 10, failed 11, canceled 0, ignored 0, pending 0 *** 11 TESTS FAILED *** mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2AppendDataStrictAnalysisSuite -Dtest=none Tests: succeeded 14, failed 11, canceled 0, ignored 0, pending 0 *** 11 TESTS FAILED *** mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2OverwriteByExpressionANSIAnalysisSuite -Dtest=none Tests: succeeded 11, failed 12, canceled 0, ignored 0, pending 0 *** 12 TESTS FAILED *** mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2OverwriteByExpressionStrictAnalysisSuite -Dtest=none Tests: succeeded 15, failed 12, canceled 0, ignored 0, pending 0 *** 12 TESTS FAILED *** mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2OverwritePartitionsDynamicANSIAnalysisSuite -Dtest=none Tests: succeeded 10, failed 11, canceled 0, ignored 0, pending 0 *** 11 TESTS FAILED *** mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2OverwritePartitionsDynamicStrictAnalysisSuite -Dtest=none Tests: succeeded 14, failed 11, canceled 0, ignored 0, pending 0 *** 11 TESTS FAILED *** mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.trees.TreeNodeSuite -Dtest=none Tests: succeeded 21, failed 1, canceled 0, ignored 0, pending 0 *** 1 TEST FAILED *** ``` **After** ``` mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2AppendDataANSIAnalysisSuite -Dtest=none Tests: succeeded 21, failed 0, canceled 0, ignored 0, pending 0 All tests passed. mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2AppendDataStrictAnalysisSuite -Dtest=none Tests: succeeded 25, failed 0, canceled 0, ignored 0, pending 0 All tests passed. mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2OverwriteByExpressionANSIAnalysisSuite -Dtest=none Tests: succeeded 23, failed 0, canceled 0, ignored 0, pending 0 All tests passed. mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2OverwriteByExpressionStrictAnalysisSuite -Dtest=none Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0 All tests passed. mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2OverwritePartitionsDynamicANSIAnalysisSuite -Dtest=none Tests: succeeded 21, failed 0, canceled 0, ignored 0, pending 0 mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.analysis.V2OverwritePartitionsDynamicStrictAnalysisSuite -Dtest=none Tests: succeeded 25, failed 0, canceled 0, ignored 0, pending 0 mvn clean test -pl sql/catalyst -am -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.catalyst.trees.TreeNodeSuite -Dtest=none Tests: succeeded 22, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` The whole 'SQL / catalyst' module test: ``` mvn clean test -pl sql/catalyst -Pscala-2.13 ``` **Before** ``` Tests: succeeded 3853, failed 103, canceled 0, ignored 6, pending 0 *** 3 SUITES ABORTED *** *** 103 TESTS FAILED *** ``` **After** ``` Tests: succeeded 3925, failed 31, canceled 0, ignored 6, pending 0 *** 3 SUITES ABORTED *** *** 31 TESTS FAILED *** ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
