Hi Sem, Nick, and all,
It is natural for users to expect a built-in DataFrame API for graph processing, just as Spark provides DataFrames alongside RDDs and spark.ml alongside the RDD-based spark.mllib. GraphFrames already provides that graph abstraction, but it remains a separate library. There is also a gap for Spark Connect users: GraphX’s RDD-based API is not supported through Connect. Bringing GraphFrames into Spark, together with contributions and expertise from the GraphFrames community, would address both needs: a built-in DataFrame-oriented graph API and graph algorithms available through Spark Connect. Thanks, Ruifeng On Mon, Nov 9, 2026 at 4:21 PM Sem <[email protected]> wrote: > Hello! > > It's difficult to say with any certainty, as it was 10 years ago. > However, from some old code comments in the GraphFrames codebase, I can > infer that the intention was to stabilize the API before merging it > with the upstream. > > Best regards, > Sem > > On Sun, 2026-11-08 at 18:16 -0500, Nicholas Chammas wrote: > > Great to see this proposal. GraphFrames has long been a very useful > > library, and I have used it many times. > > > > One historical curiosity: Why was this not originally a part of the > > project? Was it just a historical accident, or was there some > > original blocker which has since been resolved? > > > > NIck > > > > > > > On Nov 8, 2026, at 2:39 PM, Sem via dev <[email protected]> > > > wrote: > > > > > > Hello Apache Spark community, > > > > > > I would like to start a discussion about merging the GraphFrames > > > project into the Apache Spark project. > > > > > > SPIP: > > > > https://docs.google.com/document/d/1v8OEV5Qqk5LE1XID4ZiPF5rLkAk-Zz6fp0tns9Z_TA8/edit?usp=sharing > > > > > > Shepherd: Ruifeng Zheng > > > > > > Following the deprecation of GraphX, there is no public, non- > > > deprecated > > > API for working with graph-like data. At the same time, the > > > GraphFrames > > > project (https://github.com/graphframes/graphframes) provides a > > > solid > > > foundation for representing and processing connected data as > > > Spark's > > > DataFrames, as well as a rich standard library of ready-to-use > > > graph > > > algorithms such as PageRank, Connected Components and Node > > > Embeddings. > > > > > > The proposal suggests moving the GraphFrames code (Apache 2.0 > > > licence) > > > to the Apache Spark project as an in-tree module. > > > > > > Please reply to this thread. A [VOTE] will follow once the > > > discussion > > > has concluded. > > > > > > Best regards, > > > Sem > > > > > > ------------------------------------------------------------------- > > > -- > > > To unsubscribe e-mail: [email protected] > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: [email protected] >
