This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 4b83774cb14e [SPARK-50902][CORE][K8S][TESTS] Add `CRC32C` test cases
4b83774cb14e is described below
commit 4b83774cb14e6f61d44e7cbc5fade2239b889d99
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Jan 21 08:01:43 2025 -0800
[SPARK-50902][CORE][K8S][TESTS] Add `CRC32C` test cases
### What changes were proposed in this pull request?
This PR aims to add `CRC32C` test cases.
### Why are the changes needed?
Apache Spark supports `CRC32C`. We had better add more test coverage like
`CRC32`.
- #47929
### Does this PR introduce _any_ user-facing change?
No. This is a test case addition.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49582 from dongjoon-hyun/SPARK-50902.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 98f276730d1dcfb2732a78439270b2578a777a15)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/network/shuffle/ExternalBlockHandlerSuite.java | 5 +++++
.../apache/spark/shuffle/KubernetesLocalDiskShuffleDataIOSuite.scala | 2 +-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git
a/common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalBlockHandlerSuite.java
b/common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalBlockHandlerSuite.java
index 3609aa556deb..f7edc8837fde 100644
---
a/common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalBlockHandlerSuite.java
+++
b/common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalBlockHandlerSuite.java
@@ -219,6 +219,11 @@ public class ExternalBlockHandlerSuite {
checkDiagnosisResult("CRC32", Cause.CHECKSUM_VERIFY_PASS);
}
+ @Test
+ public void testShuffleCorruptionDiagnosisCRC32C() throws IOException {
+ checkDiagnosisResult("CRC32C", Cause.CHECKSUM_VERIFY_PASS);
+ }
+
@Test
public void testFetchShuffleBlocks() {
when(blockResolver.getBlockData("app0", "exec1", 0, 0,
0)).thenReturn(blockMarkers[0]);
diff --git
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleDataIOSuite.scala
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleDataIOSuite.scala
index 663be35ce0c9..25b2cad9ddd2 100644
---
a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleDataIOSuite.scala
+++
b/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleDataIOSuite.scala
@@ -230,7 +230,7 @@ class KubernetesLocalDiskShuffleDataIOSuite extends
SparkFunSuite with LocalRoot
conf.get("spark.local.dir") + "/spark-x/executor-y")
val dir = sparkConf.get("spark.local.dir") + "/blockmgr-z/00"
Files.createDirectories(new File(dir).toPath())
- Seq("ADLER32", "CRC32").foreach { algorithm =>
+ Seq("ADLER32", "CRC32", "CRC32C").foreach { algorithm =>
new File(dir, s"1.checksum.$algorithm").createNewFile()
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
