This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8593b95d9b5f [SPARK-59523][ML][FOLLOWUP] Add a compatibility test for
IDFModel saved by Spark 4.0.0
8593b95d9b5f is described below
commit 8593b95d9b5fdb53a551bf182518f839106031b0
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Thu Oct 29 08:10:44 2026 +0800
[SPARK-59523][ML][FOLLOWUP] Add a compatibility test for IDFModel saved by
Spark 4.0.0
### What changes were proposed in this pull request?
Adds a compatibility fixture and a load test for `IDFModel`:
- `mllib/src/test/resources/ml-models/idf-4.0.0`: an `IDFModel` saved by
released Spark 4.0.0 (PySpark 4.0.0 `IDF`, a 3-document, 4-term corpus with
`minDocFreq=1`).
- `IDFSuite`: "Load IDFModel saved by Spark 4.0.0" loads the fixture with
the current code and checks the IDF vector, `docFreq`, `numDocs`, and the
`sparkVersion` in the metadata.
This is a follow-up to SPARK-59523 (commit `f19cad23c2a`), which moved the
IDF data into `IDFModel`'s constructor and changed the load path. The existing
`IDFModel read/write` test only round-trips models written by the current code,
so nothing checked a model saved by a released Spark.
### Why are the changes needed?
SPARK-59523 changed how `IDFModel` is constructed and loaded. A model saved
by a released Spark should keep loading after such a change. This test catches
a regression of that kind.
### Does this PR introduce _any_ user-facing change?
No. Test and test-resource changes only.
### How was this patch tested?
Added the new test case and fixture. I did not run `IDFSuite` through sbt.
As a substitute, I compiled the current `IDF.scala` on JDK 17 with Scala 2.13
and ran a small program that calls `IDFModel.load` on the fixture. It returned
the expected IDF vector `[0.0, 0.6931471805599453, 0.28768207245178085,
0.28768207245178085]`, `docFreq` `[0, 1, 2, 2]`, `numDocs` 3, and
`sparkVersion` 4.0.0. The expected values come from the PySpark 4.0.0 run that
wrote the fixture.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (system.ai.claude-haiku-5-5) via Claude Code
This pull request and its description were written by Isaac.
Closes #59864 from zhengruifeng/idf-model-fixture-compat.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../resources/ml-models/idf-4.0.0/data/._SUCCESS.crc | Bin 0 -> 8 bytes
...2-5754-4f77-b2f8-049d54d7bf63-c000.snappy.parquet.crc | Bin 0 -> 28 bytes
.../src/test/resources/ml-models/idf-4.0.0/data/_SUCCESS | 0
...cdfa2-5754-4f77-b2f8-049d54d7bf63-c000.snappy.parquet | Bin 0 -> 2350 bytes
.../resources/ml-models/idf-4.0.0/metadata/._SUCCESS.crc | Bin 0 -> 8 bytes
...000-d6d075e0-221c-4638-a374-e4a8d0739994-c000.txt.crc | Bin 0 -> 12 bytes
.../test/resources/ml-models/idf-4.0.0/metadata/_SUCCESS | 0
...t-00000-d6d075e0-221c-4638-a374-e4a8d0739994-c000.txt | 1 +
.../scala/org/apache/spark/ml/feature/IDFSuite.scala | 14 ++++++++++++++
9 files changed, 15 insertions(+)
diff --git a/mllib/src/test/resources/ml-models/idf-4.0.0/data/._SUCCESS.crc
b/mllib/src/test/resources/ml-models/idf-4.0.0/data/._SUCCESS.crc
new file mode 100644
index 000000000000..3b7b044936a8
Binary files /dev/null and
b/mllib/src/test/resources/ml-models/idf-4.0.0/data/._SUCCESS.crc differ
diff --git
a/mllib/src/test/resources/ml-models/idf-4.0.0/data/.part-00000-de7cdfa2-5754-4f77-b2f8-049d54d7bf63-c000.snappy.parquet.crc
b/mllib/src/test/resources/ml-models/idf-4.0.0/data/.part-00000-de7cdfa2-5754-4f77-b2f8-049d54d7bf63-c000.snappy.parquet.crc
new file mode 100644
index 000000000000..3015831fa4ee
Binary files /dev/null and
b/mllib/src/test/resources/ml-models/idf-4.0.0/data/.part-00000-de7cdfa2-5754-4f77-b2f8-049d54d7bf63-c000.snappy.parquet.crc
differ
diff --git a/mllib/src/test/resources/ml-models/idf-4.0.0/data/_SUCCESS
b/mllib/src/test/resources/ml-models/idf-4.0.0/data/_SUCCESS
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git
a/mllib/src/test/resources/ml-models/idf-4.0.0/data/part-00000-de7cdfa2-5754-4f77-b2f8-049d54d7bf63-c000.snappy.parquet
b/mllib/src/test/resources/ml-models/idf-4.0.0/data/part-00000-de7cdfa2-5754-4f77-b2f8-049d54d7bf63-c000.snappy.parquet
new file mode 100644
index 000000000000..1e03b53a0a12
Binary files /dev/null and
b/mllib/src/test/resources/ml-models/idf-4.0.0/data/part-00000-de7cdfa2-5754-4f77-b2f8-049d54d7bf63-c000.snappy.parquet
differ
diff --git
a/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/._SUCCESS.crc
b/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/._SUCCESS.crc
new file mode 100644
index 000000000000..3b7b044936a8
Binary files /dev/null and
b/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/._SUCCESS.crc differ
diff --git
a/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/.part-00000-d6d075e0-221c-4638-a374-e4a8d0739994-c000.txt.crc
b/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/.part-00000-d6d075e0-221c-4638-a374-e4a8d0739994-c000.txt.crc
new file mode 100644
index 000000000000..c188d4c84e04
Binary files /dev/null and
b/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/.part-00000-d6d075e0-221c-4638-a374-e4a8d0739994-c000.txt.crc
differ
diff --git a/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/_SUCCESS
b/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/_SUCCESS
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git
a/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/part-00000-d6d075e0-221c-4638-a374-e4a8d0739994-c000.txt
b/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/part-00000-d6d075e0-221c-4638-a374-e4a8d0739994-c000.txt
new file mode 100644
index 000000000000..965c55e31ceb
--- /dev/null
+++
b/mllib/src/test/resources/ml-models/idf-4.0.0/metadata/part-00000-d6d075e0-221c-4638-a374-e4a8d0739994-c000.txt
@@ -0,0 +1 @@
+{"class":"org.apache.spark.ml.feature.IDFModel","timestamp":1793181639444,"sparkVersion":"4.0.0","uid":"IDF_00037a2dc04f","paramMap":{"minDocFreq":1,"outputCol":"scaled","inputCol":"features"},"defaultParamMap":{"minDocFreq":0,"outputCol":"IDF_00037a2dc04f__output"}}
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/IDFSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/feature/IDFSuite.scala
index 88082a4209bf..8ae8122d0f74 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/feature/IDFSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/feature/IDFSuite.scala
@@ -139,4 +139,18 @@ class IDFSuite extends MLTest with DefaultReadWriteTest {
assert(newInstance.numDocs === instance.numDocs)
}
}
+
+ test("Load IDFModel saved by Spark 4.0.0") {
+ val modelPath = testFile("ml-models/idf-4.0.0")
+
+ val loadedModel = IDFModel.load(modelPath)
+ assert(loadedModel.idf === Vectors.dense(
+ 0.0, 0.6931471805599453, 0.28768207245178085, 0.28768207245178085))
+ assert(loadedModel.docFreq === Array(0L, 1L, 2L, 2L))
+ assert(loadedModel.numDocs === 3L)
+
+ val metadata = spark.read.json(s"$modelPath/metadata")
+ val sparkVersionStr = metadata.select("sparkVersion").first().getString(0)
+ assert(sparkVersionStr === "4.0.0")
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
