(incubator-uniffle) branch master updated: [#1591] feat(spark): Support Spark 3.5.1 (#1592)

[roryqi]

This is an automated email from the ASF dual-hosted git repository.

roryqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git


The following commit(s) were added to refs/heads/master by this push:
     new dd67774e1 [#1591] feat(spark): Support Spark 3.5.1 (#1592)
dd67774e1 is described below

commit dd67774e106c902110b17fc1d6f23937f70a8fe7
Author: QI Jiale <qijial...@foxmail.com>
AuthorDate: Wed Mar 20 14:22:55 2024 +0800

    [#1591] feat(spark): Support Spark 3.5.1 (#1592)
    
    ### What changes were proposed in this pull request?
    
    Support Spark 3.5.1
    
    The abstract class `TaskContext` introduced a new method `isFailed()` after 
Spark version 3.5.1. Due to this addition, it is not possible to compile the 
`MockTaskContext` class using Spark 3.5.1 without modifying the code. 
Consequently, we also added the `isFailed()` method to `MockTaskContext`. 
However, to maintain compatibility with versions of Spark prior to 3.5.1, we 
did not annotate this method with `@Override`.
    
    Compiling old code with Spark 3.5.1 results in the following error:
    ```error: MockTaskContext is not abstract and does not override abstract 
method isFailed() in TaskContext.```
    Additionally, adding the `@Override` annotation leads to another error:
    ```error: method does not override or implement a method from a 
supertype.```
    
    ### Why are the changes needed?
    
    Fix: #1591
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Existing tests.
---
 .../spark3/src/test/java/org/apache/uniffle/test/GetReaderTest.java | 6 ++++++
 pom.xml                                                             | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git 
a/integration-test/spark3/src/test/java/org/apache/uniffle/test/GetReaderTest.java
 
b/integration-test/spark3/src/test/java/org/apache/uniffle/test/GetReaderTest.java
index 32deea322..e1b5dc037 100644
--- 
a/integration-test/spark3/src/test/java/org/apache/uniffle/test/GetReaderTest.java
+++ 
b/integration-test/spark3/src/test/java/org/apache/uniffle/test/GetReaderTest.java
@@ -297,6 +297,12 @@ public class GetReaderTest extends IntegrationTestBase {
       return false;
     }
 
+    // The following method is only available after Spark 3.5.1, and in order 
to be compatible
+    // with the version before Spark 3.5.1, the annotation @Override is not 
added.
+    public boolean isFailed() {
+      return false;
+    }
+
     @Override
     public Properties getLocalProperties() {
       return null;
diff --git a/pom.xml b/pom.xml
index 2491416f2..69dc4d85d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1761,7 +1761,7 @@
       <id>spark3.5</id>
       <properties>
         <scala.binary.version>2.12</scala.binary.version>
-        <spark.version>3.5.0</spark.version>
+        <spark.version>3.5.1</spark.version>
         <client.type>3</client.type>
         <jackson.version>2.15.2</jackson.version>
         <log4j.version>2.20.0</log4j.version>