Kalvin2077 commented on code in PR #3834:
URL: https://github.com/apache/celeborn/pull/3834#discussion_r4004264954
##########
client-spark/spark-3/src/test/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleManagerSuite.scala:
##########
@@ -92,6 +92,20 @@ class SparkShuffleManagerSuite extends Logging {
sc.stop()
}
+ @junit.Test
+ def testReliableStorageFollowsFallbackPolicy(): Unit = {
+ // supportsReliableStorage must be true only for NEVER; AUTO/ALWAYS may
fall back to local-disk
+ // shuffle, so reporting reliable storage would let DRA reclaim executors
holding fallback data.
+ def reliable(policy: String): Boolean = {
+ val conf = new SparkConf()
+ .set(s"spark.${CelebornConf.SPARK_SHUFFLE_FALLBACK_POLICY.key}",
policy)
+ new CelebornShuffleDataIO(conf).driver().supportsReliableStorage()
Review Comment:
**[P1] New test calls the Spark 3.5 interface, causing older versions of
Spark to fail compilation**
The declared return type is Spark’s `ShuffleDriverComponents`, and this
interface does not have `supportsReliableStorage()` in Spark 3.0–3.4. This
shared test therefore fails during the test compilation phase for older
profiles, and this cannot be avoided even when configured to use
`LocalDiskShuffleDataIO`. When performing a minimal Scala compilation using the
actual JARs for Spark 3.0.3 and 3.4.4, the error `value supportsReliableStorage
is not a member of ... ShuffleDriverComponents`; the corresponding test for
3.5.1 passed. It is recommended to cast the return value to
`CelebornShuffleDriverComponents` (from the same package) and call its existing
methods, or use a compatible reflection-based approach, and then re-verify the
test compilation for Spark 3.0–3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
