Interesting, personally there are many use cases where I would recommend RDDs — definitely to more advanced users — and I think that RDDs and GraphX are in pretty different boats (RDDs are very actively used).
Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her On Fri, Oct 4, 2024 at 3:08 PM Mark Hamstra <markhams...@gmail.com> wrote: > No, I wouldn't encourage anyone to base a new production deployment on > GraphX, but neither would I encourage new production deployments based > on the RDD API without deep study and understanding of the > implications and limitations. What I would be most comfortable with is > documenting the current status and shortcomings of GraphX, along with > encouraging contributions to remedy that situation. I'm not sure what > the best (or even an effective) way of accomplishing that is, but I'm > pretty sure it's not just labeling GraphX as deprecated. > > On Fri, Oct 4, 2024 at 3:00 PM Holden Karau <holden.ka...@gmail.com> > wrote: > > > > Personally I think people should not depend on it — there’s literally no > one working on it, and not being up front about that I think draws > everything else into question. > > > > Would anyone here feel comfortable using GraphX for a new production > deployment today? > > > > > > Twitter: https://twitter.com/holdenkarau > > Fight Health Insurance: https://www.fighthealthinsurance.com/ > > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > Pronouns: she/her > > > > > > On Fri, Oct 4, 2024 at 2:56 PM Mark Hamstra <markhams...@gmail.com> > wrote: > >> > >> I'm -1(*) because, while it technically means "might be removed in the > >> future", I think developers and users are more prone to interpret > >> something being marked as deprecated as "very likely will be removed > >> in the future, so don't depend on this or waste your time contributing > >> to its further development." I don't think the latter is what we want > >> just because something hasn't been updated meaningfully in a while. > >> There have been How To articles for GraphX and Graph Frames posted in > >> the not too distant past, and the Google Search trend shows a pretty > >> steady level of interest, not a decline to zero, so I don't think that > >> it is accurate to declare that there is no use or interest in GraphX. > >> > >> Unless retaining GraphX is imposing significant costs on continuing > >> Spark development, I can't support deprecating GraphX. I can support > >> encouraging GraphX and Graph Frames development through something like > >> a To Do list or document of "What we'd like to see in the way of > >> further development of Spark's graph processing capabilities" -- i.e., > >> things that encourage and support new contributions to address any > >> shortcomings in Spark's graph processing, not things that discourage > >> contributions and use in the way that I believe simply declaring > >> GraphX to be deprecated would. > >> > >> > >> On Sun, Sep 29, 2024 at 11:04 AM Holden Karau <holden.ka...@gmail.com> > wrote: > >> > > >> > Since we're getting close to cutting a 4.0 branch I'd like to float > the idea of officially deprecating Graph X. What that would mean (to me) is > we would update the docs to indicate that Graph X is deprecated and it's > APIs may be removed at anytime in the future. > >> > > >> > Alternatively, we could mark it as "unmaintained and in search of > maintainers" with a note that if no maintainers are found, we may remove it > in a future minor version. > >> > > >> > Looking at the source graph X, I don't see any meaningful active > development going back over three years*. There is even a thread on user@ > from 2017 asking if graph X is maintained anymore, with no response from > the developers. > >> > > >> > Now I'm open to the idea that GraphX is stable and "works as is" and > simply doesn't require modifications but given the user thread I'm a little > concerned here about bringing this API with us into Spark 4 if we don't > have anyone signed up to maintain it. > >> > > >> > * Excluding globally applied changes > >> > -- > >> > Twitter: https://twitter.com/holdenkarau > >> > Fight Health Insurance: https://www.fighthealthinsurance.com/ > >> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > >> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > >> > Pronouns: she/her > >> > >> --------------------------------------------------------------------- > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >> >
