zhengruifeng opened a new pull request, #58422: URL: https://github.com/apache/spark/pull/58422
### What changes were proposed in this pull request? This exploratory draft ports the material runtime differences from GraphFrames' namespaced GraphX fork back onto canonical GraphX so that the changes can be reviewed directly: - use vector messages in label propagation and resolve equal-frequency labels deterministically by choosing the smallest label; - remove five implicit `cache()` calls from selected graph construction, aggregation, and join paths; - retain Pregel's message persistence management but disable periodic message-RDD checkpoints; - add directed and undirected shortest paths while preserving the existing two-argument JVM method; - explicitly reject unknown `EdgeDirection` values in three match expressions. It also adds focused label-propagation, shortest-path, and persistence tests, and updates the Pregel checkpoint documentation. This does not vendor GraphFrames' GraphX copy. It deliberately retains canonical Spark's logging and newer SVD++ correctness fix, and leaves GraphFrames-specific conversion/unpersist handling in #58302. Provenance: - GraphX copy: https://github.com/graphframes/graphframes/pull/680 - label propagation: https://github.com/graphframes/graphframes/pull/681 - persistence/checkpoint changes: https://github.com/graphframes/graphframes/pull/687 - shortest-path direction: https://github.com/graphframes/graphframes/pull/737 ### Why are the changes needed? This is a companion prototype for #58302. Its purpose is to make the private-fork delta visible and independently reviewable; it is not a claim that all of these changes should merge together as-is. Important questions remain: - vector label messages retain every neighbor label and violate the stated commutative merge contract, so their shuffle/memory behavior needs benchmarking against the current count maps; - removing implicit persistence can prevent retained-RDD leaks but can also increase recomputation and changes observable storage behavior; - graph checkpointing does not obviously truncate the separate active-message dependency, so disabling message checkpoints needs long-lineage validation before merge. ### Does this PR introduce _any_ user-facing change? Yes. It adds an undirected shortest-path overload, makes label-propagation ties deterministic, changes implicit GraphX persistence, and changes `spark.graphx.pregel.checkpointInterval` to checkpoint only graph RDDs. ### How was this patch tested? - `./build/sbt graphx/test` (113 tests passed) - `./dev/lint-scala` - `git diff --check` The Pregel checkpoint change does not yet have a direct long-lineage regression test; that is intentionally called out as follow-up work for this exploratory draft. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
