zhengruifeng opened a new pull request, #58302:
URL: https://github.com/apache/spark/pull/58302
### What changes were proposed in this pull request?
This draft starts an incremental upstreaming of GraphFrames into Apache
Spark. It makes the
module and API shape concrete so that the Spark and GraphFrames communities
can review the
architecture before algorithms are moved.
The patch:
- adds a top-level `spark-graphframes` module and includes its JAR in Spark
distributions;
- introduces the JVM API under `org.apache.spark.graphframes`;
- introduces the Python API under `pyspark.graphframes`;
- implements graph construction, validation, vertices, edges, triplets,
degree DataFrames,
filtering, isolated-vertex removal, reversal, and undirected conversion;
- implements the Python surface with built-in DataFrame operations so this
initial API works in
both classic Spark and Spark Connect; and
- adds Scala, classic PySpark, and Spark Connect tests.
This intentionally does not import GraphFrames' modified GraphX fork. Motif
finding,
AggregateMessages, Pregel, BFS, connected components, shortest paths,
PageRank, other algorithms,
compatibility artifacts, documentation, and benchmarks remain follow-up work.
This is a draft and is not proposed for merge until the community agrees on
the SPIP, initial
feature scope, package/artifact compatibility, maintainership, and any
required IP-clearance
process. The working upstreaming plan is here:
https://docs.google.com/document/d/1emar4QUbrf30jC1Iffv41AVBr-UmRoz2H32VRG3zmAs/edit
### Why are the changes needed?
GraphFrames provides a widely used DataFrame graph abstraction, but its
external release model
requires a separate Spark/Scala compatibility matrix and separately packaged
Spark Connect
extensions. An in-tree module could provide release-aligned Scala, Java,
Python, and Connect APIs
and allow graph operations to evolve with Spark SQL.
Starting with a GraphX-free relational core establishes module boundaries
and public namespaces
without committing Spark to a duplicate deprecated graph runtime or
importing all algorithms in a
single unreviewable change.
### Does this PR introduce _any_ user-facing change?
Yes. It proposes experimental APIs under new Spark-owned namespaces:
```scala
import org.apache.spark.graphframes.GraphFrame
val graph = GraphFrame(vertices, edges)
graph.degrees.show()
```
```python
from pyspark.graphframes import GraphFrame
graph = GraphFrame(vertices, edges)
graph.degrees.show()
```
No released Spark version contains these APIs. Existing `org.graphframes`
and `graphframes`
packages are not changed by this patch.
### How was this patch tested?
- `build/sbt graphframes/Test/compile`
- `build/sbt 'graphframes/testOnly
org.apache.spark.graphframes.GraphFrameSuite'`
- `build/sbt -Phive assembly/package`
- `python/run-tests --testnames pyspark.graphframes.tests.test_graphframe`
- `python/run-tests --testnames
pyspark.graphframes.tests.connect.test_parity_graphframe`
- `build/sbt graphframes/scalastyle graphframes/Test/scalastyle`
- `dev/lint-python --compile --ruff`
- `dev/check-license`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
