zhengruifeng opened a new pull request, #59864: URL: https://github.com/apache/spark/pull/59864
### What changes were proposed in this pull request? Adds a compatibility fixture and a load test for `IDFModel`: - `mllib/src/test/resources/ml-models/idf-4.0.0`: an `IDFModel` saved by released Spark 4.0.0 (PySpark 4.0.0 `IDF`, a 3-document, 4-term corpus with `minDocFreq=1`). - `IDFSuite`: "Load IDFModel saved by Spark 4.0.0" loads the fixture with the current code and checks the IDF vector, `docFreq`, `numDocs`, and the `sparkVersion` in the metadata. This is a follow-up to SPARK-59523 (commit `f19cad23c2a`), which moved the IDF data into `IDFModel`'s constructor and changed the load path. The existing `IDFModel read/write` test only round-trips models written by the current code, so nothing checked a model saved by a released Spark. ### Why are the changes needed? SPARK-59523 changed how `IDFModel` is constructed and loaded. A model saved by a released Spark should keep loading after such a change. This test catches a regression of that kind. ### Does this PR introduce _any_ user-facing change? No. Test and test-resource changes only. ### How was this patch tested? Added the new test case and fixture. I did not run `IDFSuite` through sbt. As a substitute, I compiled the current `IDF.scala` on JDK 17 with Scala 2.13 and ran a small program that calls `IDFModel.load` on the fixture. It returned the expected IDF vector `[0.0, 0.6931471805599453, 0.28768207245178085, 0.28768207245178085]`, `docFreq` `[0, 1, 2, 2]`, `numDocs` 3, and `sparkVersion` 4.0.0. The expected values come from the PySpark 4.0.0 run that wrote the fixture. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (system.ai.claude-haiku-5-5) via Claude Code This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
