LuciferYang opened a new pull request, #41654:
URL: https://github.com/apache/spark/pull/41654
### What changes were proposed in this pull request?
This pr add a new `apply` function to `NonFateSharingCache` and change
`CodeGenerator` to use the new `NonFateSharingCache#apply` to make `catalyst`
module can test using maven.
### Why are the changes needed?
SPARK-43300 introduced `NonFateSharingCache` to `core` module and it only
used by `CodeGenerator` which is in `catalyst` module.
There are two `apply` funcitons in `NonFateSharingCache` and the input
parameter of `NonFateSharingCache#apply` is
`com.google.common.cache.Cache/LoadingCache`.
The `catalyst` module may use shaded spark-core jar when we do maven testing
and in the shaded spark-core jar, the Guava related classes will be relocated
from `com.google.common` to `org.sparkproject.guava`, so the input parameter of
`NonFateSharingCache#apply` will change to
`org.sparkproject.guava.cache.Cache/LoadingCache`, but the catalyst module has
not been shaded yet when do maven testing, so `CodeGenerator` will still use
type `com.google.common.cache.Cache` to call the `NonFateSharingCache#apply`
function, then this will result in a mismatch of input types when do maven
testing and maven test will aborted as follows:
```
ProductAggSuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$
at
org.apache.spark.sql.catalyst.expressions.codegen.JavaCode$.variable(javaCode.scala:64)
at
org.apache.spark.sql.catalyst.expressions.codegen.JavaCode$.isNullVariable(javaCode.scala:77)
at
org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:200)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:196)
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.$anonfun$create$1(GenerateSafeProjection.scala:156)
at scala.collection.immutable.List.map(List.scala:293)
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:153)
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:39)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:1369)
```
So this pr add a new `apply` function to break non-core modules directly
using Guava Cache related types as input parameter to invoke
`NonFateSharingCache#apply` function for workaround, this way can avoid
non-core modules Maven test failures caused by using shaded core module.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual test
```
build/mvn clean install -DskipTests -pl sql/catalyst -am
build/mvn test -pl sql/catalyst
```
Before
```
ProductAggSuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$
at
org.apache.spark.sql.catalyst.expressions.codegen.JavaCode$.variable(javaCode.scala:64)
at
org.apache.spark.sql.catalyst.expressions.codegen.JavaCode$.isNullVariable(javaCode.scala:77)
at
org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:200)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:196)
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.$anonfun$create$1(GenerateSafeProjection.scala:156)
at scala.collection.immutable.List.map(List.scala:293)
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:153)
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:39)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:1369)
```
After
```
Run completed in 4 minutes, 22 seconds.
Total number of tests run: 7088
Suites: completed 295, aborted 0
Tests: succeeded 7088, failed 0, canceled 0, ignored 5, pending 0
All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
