LuciferYang commented on code in PR #48818:
URL: https://github.com/apache/spark/pull/48818#discussion_r1929652392
##########
sql/connect/shims/src/main/scala/org/apache/spark/shims.scala:
##########
@@ -35,6 +35,7 @@ package sql {
package execution {
class QueryExecution
+ class SparkStrategy
Review Comment:
This PR seems to have caused the Maven daily tests to fail:
- 4.0 & Java 17:
https://github.com/apache/spark/actions/runs/12965584663/job/36165484414
- master & Java 17:
https://github.com/apache/spark/actions/runs/12965181274/job/36164647472
- master & Java 21:
https://github.com/apache/spark/actions/runs/12965580081/job/36165475388
I did some local testing with both Maven and SBT, and they both failed the
tests:
- maven
before:
```
git reset --hard a03c4cbd37f0409469cb9b5089f70612b78a27eb // before this
one: [SPARK-50973][SQL][AVRO] Cleanup deprecated api usage related to
`avro.Schema#toString(boolean)`
build/mvn clean test -pl sql/catalyst -am
-DwildcardSuites=org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite
-Dtest=none
```
```
Run completed in 2 seconds, 973 milliseconds.
Total number of tests run: 28
Suites: completed 2, aborted 0
Tests: succeeded 28, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
after:
```
git reset --hard 5db31aec33c53aaa7c814f33ec84e6ba66fc193b // this one:
[SPARK-49700][CONNECT][SQL] Unified Scala Interface for Connect and Classic
build/mvn clean test -pl sql/catalyst -am
-DwildcardSuites=org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite
-Dtest=none
```
```
EncoderResolutionSuite:
org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite *** ABORTED ***
java.lang.RuntimeException: Unable to load a Suite class
org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite that was
discovered in the runpath: error reading Scala signature of
org.apache.spark.sql.package: unsafe symbol SparkStrategy (child of package
execution) in runtime reflection universe
at
org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:80)
at
org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
at scala.collection.immutable.Vector1.map(Vector.scala:2141)
at scala.collection.immutable.Vector1.map(Vector.scala:386)
at org.scalatest.tools.DiscoverySuite.<init>(DiscoverySuite.scala:37)
at
org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$3(Runner.scala:1140)
at scala.collection.immutable.List.map(List.scala:247)
at org.scalatest.tools.Runner$.genDiscoSuites$1(Runner.scala:1139)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1225)
at
org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
...
Cause: java.lang.RuntimeException: error reading Scala signature of
org.apache.spark.sql.package: unsafe symbol SparkStrategy (child of package
execution) in runtime reflection universe
at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:48)
at
scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:675)
at
scala.reflect.runtime.SymbolLoaders$TopClassCompleter.$anonfun$complete$3(SymbolLoaders.scala:37)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at
scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:321)
at
scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:34)
at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1592)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1548)
at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$13.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:221)
at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol.info(SynchronizedSymbols.scala:158)
...
```
- sbt
before:
```
git reset --hard a03c4cbd37f0409469cb9b5089f70612b78a27eb // before this
one: [SPARK-50973][SQL][AVRO] Cleanup deprecated api usage related to
`avro.Schema#toString(boolean)`
build/sbt clean "catalyst/testOnly
org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite"
```
```
[info] Run completed in 2 seconds, 160 milliseconds.
[info] Total number of tests run: 28
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 28, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
after:
```
git reset --hard 5db31aec33c53aaa7c814f33ec84e6ba66fc193b // this one:
[SPARK-49700][CONNECT][SQL] Unified Scala Interface for Connect and Classic
build/sbt clean "catalyst/testOnly
org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite"
```
```
[info] EncoderResolutionSuite:
[info] org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite ***
ABORTED *** (13 milliseconds)
[info] java.lang.RuntimeException: error reading Scala signature of
org.apache.spark.sql.package: unsafe symbol SparkStrategy (child of package
execution) in runtime reflection universe
[info] at
scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:48)
[info] at
scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:675)
[info] at
scala.reflect.runtime.SymbolLoaders$TopClassCompleter.$anonfun$complete$3(SymbolLoaders.scala:37)
[info] at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[info] at
scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:321)
[info] at
scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:34)
[info] at
scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1592)
[info] at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1548)
[info] at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$13.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:221)
[info] at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol.info(SynchronizedSymbols.scala:158)
[info] at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol.info$(SynchronizedSymbols.scala:158)
[info] at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$13.info(SynchronizedSymbols.scala:221)
[info] at
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:345)
[info] at
scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:399)
[info] at
scala.reflect.runtime.SymbolLoaders$LazyPackageType.$anonfun$complete$5(SymbolLoaders.scala:83)
[info] at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[info] at
scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:321)
[info] at
scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:80)
[info] at
scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1592)
[info] at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1548)
[info] at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$9.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:209)
[info] at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol.info(SynchronizedSymbols.scala:158)
[info] at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol.info$(SynchronizedSymbols.scala:158)
[info] at
scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$9.info(SynchronizedSymbols.scala:209)
[info] at
scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2605)
[info] at
scala.reflect.internal.tpe.FindMembers$FindMemberBase.init(FindMembers.scala:37)
[info] at
scala.reflect.internal.tpe.FindMembers$FindMember.init(FindMembers.scala:258)
[info] at
scala.reflect.internal.Types$Type.$anonfun$findMember$1(Types.scala:1031)
[info] at
scala.reflect.internal.Types$Type.findMemberInternal$1(Types.scala:1030)
[info] at scala.reflect.internal.Types$Type.findMember(Types.scala:1035)
[info] at
scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:661)
[info] at scala.reflect.internal.Types$Type.member(Types.scala:625)
[info] at
scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:57)
[info] at
scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:29)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection$$typecreator9$1.apply(ScalaReflection.scala:305)
[info] at
scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:238)
[info] at
scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:238)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection.localTypeOf(ScalaReflection.scala:458)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection.localTypeOf$(ScalaReflection.scala:456)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection$.localTypeOf(ScalaReflection.scala:42)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection$.encoderFor(ScalaReflection.scala:305)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection$.$anonfun$encoderFor$1(ScalaReflection.scala:255)
[info] at
scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:73)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects(ScalaReflection.scala:442)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects$(ScalaReflection.scala:441)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:42)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection$.encoderFor(ScalaReflection.scala:252)
[info] at
org.apache.spark.sql.catalyst.ScalaReflection$.encoderFor(ScalaReflection.scala:236)
[info] at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:52)
[info] at
org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite.castSuccess(EncoderResolutionSuite.scala:306)
[info] at
org.apache.spark.sql.catalyst.encoders.EncoderResolutionSuite.<init>(EncoderResolutionSuite.scala:292)
[info] at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
[info] at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
[info] at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info] at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
[info] at
java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[info] at
java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347)
[info] at java.base/java.lang.Class.newInstance(Class.java:645)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Run completed in 1 second, 148 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 1
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] *** 1 SUITE ABORTED ***
```
Perhaps this is just a compilation-related issue? Do you have time to take a
look at this issue? @hvanhovell
also cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
