I'm looking at Spark's JIRA on a search for GraphX and I thought I would ask rather than just slog through it: anyone got some low hanging fruit bugs they can suggest I fix?
Thanks, Russell On Thu, Nov 14, 2024 at 11:49 AM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > + 1 > > Mich Talebzadeh, > > Architect | Data Engineer | Data Science | Financial Crime > PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College > London <https://en.wikipedia.org/wiki/Imperial_College_London> > London, United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Thu, 14 Nov 2024 at 18:52, Russell Jurney <russell.jur...@gmail.com> > wrote: > >> Okay, first I’m going to fix a bug or two, I’ll get started on an SPIP. >> >> Russ >> >> On Wed, Nov 13, 2024 at 1:56 PM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> Hm. Since it sounds like a plan why Russell you go ahead and create a >>> SPIP for it, then, this discussion takes a formal approach and is >>> documented. Otherwise we are just flogging a dead horse so to speak. >>> >>> HTH >>> >>> Mich Talebzadeh, >>> >>> Architect | Data Engineer | Data Science | Financial Crime >>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>> College London <https://en.wikipedia.org/wiki/Imperial_College_London> >>> London, United Kingdom >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>> >>> >>> On Wed, 13 Nov 2024 at 20:10, Russell Jurney <russell.jur...@gmail.com> >>> wrote: >>> >>>> It might be, but graph processing is a desirable, very useful feature >>>> of Spark. GraphX doesn't see more popularity because it never got a >>>> DataFrame interface. If someone is willing to add one and maintain it, that >>>> seems best of all. >>>> >>>> Russ >>>> >>>> On Wed, Nov 13, 2024 at 7:12 AM Ángel <angel.alvarez.pas...@gmail.com> >>>> wrote: >>>> >>>>> Seems to me.... it would be easier to move GraphX to graphframes than >>>>> the opposite. >>>>> >>>>> El mar, 8 oct 2024 a las 21:52, Reynold Xin >>>>> (<r...@databricks.com.invalid>) escribió: >>>>> >>>>>> We can also consider the following: move GraphFrame into Spark, and >>>>>> make GraphX an internal impl detail of GraphFrame. Then we can over time >>>>>> change the implementation, simplify it (not sure if it is possible, but >>>>>> somebody can look into it).... >>>>>> >>>>>> On Mon, Oct 7, 2024 at 7:04 PM Russell Jurney < >>>>>> russell.jur...@gmail.com> wrote: >>>>>> >>>>>>> Took a look at recent activity. Spark 3.5 support >>>>>>> <https://github.com/graphframes/graphframes/commit/e54f249605dde60787f9b41b88ed7d5872b7dfab> >>>>>>> was >>>>>>> added a year ago. I'm sure we'll add Spark 4 support as soon as it is >>>>>>> out. >>>>>>> >>>>>>> There is a new issue to organize a GraphFrames Hackathon >>>>>>> <https://github.com/graphframes/graphframes/issues/460>. Please >>>>>>> sign up to help! >>>>>>> https://github.com/graphframes/graphframes/issues/460 >>>>>>> >>>>>>> I seriously need GraphX and GraphFrames to make it... I have no >>>>>>> other way of doing property graph motif matching on large graphs. It's >>>>>>> kind >>>>>>> of important to me. >>>>>>> >>>>>>> Some slides on my work with GraphFrames: >>>>>>> >>>>>>> [image: image.png] >>>>>>> >>>>>>> [image: image.png] >>>>>>> >>>>>>> [image: image.png] >>>>>>> >>>>>>> [image: image.png] >>>>>>> >>>>>>> [image: image.png] >>>>>>> >>>>>>> Russell >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 7, 2024 at 6:06 PM Holden Karau <holden.ka...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> That’s awesome! >>>>>>>> >>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>> Pronouns: she/her >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 7, 2024 at 5:42 PM Russell Jurney < >>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I’ll organize a hackathon. A friend wants to finish the >>>>>>>>> implementation of Lucian modularity for GraphFrames. I’ll fix some >>>>>>>>> GraphX >>>>>>>>> bugs at it. >>>>>>>>> >>>>>>>>> I did just blog all about the motif matching in GraphFrames: >>>>>>>>> >>>>>>>>> https://blog.graphlet.ai/financial-crime-and-corruption-network-motifs-4cf2e8e10eb5 >>>>>>>>> >>>>>>>>> Russ >>>>>>>>> >>>>>>>>> On Mon, Oct 7, 2024 at 5:38 PM Holden Karau < >>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> So this discuss thread and the vote thread to deprecate to leave >>>>>>>>>> the option of removing it during 4.X are probably the highest >>>>>>>>>> profile it’s >>>>>>>>>> been in years. >>>>>>>>>> >>>>>>>>>> In the past for parts of Spark I’ve cared about I’ve organized >>>>>>>>>> virtual meetings to co-ordinate work — if your connected with some >>>>>>>>>> of the >>>>>>>>>> Spark+Graph community reaching out to find others and organizing a >>>>>>>>>> meeting >>>>>>>>>> could be a way to raise the profile a bit? Maybe organize a virtual >>>>>>>>>> hackathon (I’m meaning to try this for some other things so happy to >>>>>>>>>> share >>>>>>>>>> what I learn from doing that)? >>>>>>>>>> >>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>> Pronouns: she/her >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Oct 7, 2024 at 5:02 PM Russell Jurney < >>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> I’ll look for a bug to fix. If GraphX is outside of Spark, Spark >>>>>>>>>>> would tend to break GraphFrames and it will be burdensome on an >>>>>>>>>>> external >>>>>>>>>>> project to keep up. Graph computing on Spark is implrtant to a lot >>>>>>>>>>> of >>>>>>>>>>> people, is there a way to raise visibility here? >>>>>>>>>>> >>>>>>>>>>> On Mon, Oct 7, 2024 at 4:24 PM Holden Karau < >>>>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> There are no specific tickets associated with the lack of >>>>>>>>>>>> maintaince or this as the component has not been maintained for a >>>>>>>>>>>> sufficiently long time. If your interested in taking it on that’s >>>>>>>>>>>> wonderful, probably starting with fixing some bugs could be a >>>>>>>>>>>> great place >>>>>>>>>>>> to start and figure out if it’s something you want to do long term. >>>>>>>>>>>> >>>>>>>>>>>> I would recommend making a first bug fix in a actively >>>>>>>>>>>> maintained area of Spark to get to >>>>>>>>>>>> Know some reviewers since there is not anyone tracking the >>>>>>>>>>>> GraphX PRs. >>>>>>>>>>>> >>>>>>>>>>>> As a note I don’t think GraphX is required for Graph Frames >>>>>>>>>>>> long term, so another option would be to talk to the GraphFrames >>>>>>>>>>>> folks and >>>>>>>>>>>> move the GraphX code over to it. >>>>>>>>>>>> >>>>>>>>>>>> Ideally we’d have someone willing to act as a mentor or guide >>>>>>>>>>>> but so far we have no volunteers (especially no one familiar with >>>>>>>>>>>> the graph >>>>>>>>>>>> X code). >>>>>>>>>>>> >>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>>>> Pronouns: she/her >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Oct 7, 2024 at 3:25 PM Russell Jurney < >>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I volunteer to maintain GraphX to keep GraphFrames a viable >>>>>>>>>>>>> project. I don’t have a clear view on whether it works with Spark >>>>>>>>>>>>> 4 or if >>>>>>>>>>>>> it needs updates? I don’t have Spark commits but I’m a committer >>>>>>>>>>>>> on Apache >>>>>>>>>>>>> DataFu and mentored the Spark feature for it. >>>>>>>>>>>>> >>>>>>>>>>>>> Can someone tell me what is involved? Point me at a ticket? >>>>>>>>>>>>> >>>>>>>>>>>>> Russell >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Oct 7, 2024 at 12:11 AM Erik Eklund < >>>>>>>>>>>>> eekl...@definitivehc.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> We rely on GraphX for an important component of our product. >>>>>>>>>>>>>> And we really want it to stay a typed interface. Please keep >>>>>>>>>>>>>> GraphX. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Erik >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *From: *Holden Karau <holden.ka...@gmail.com> >>>>>>>>>>>>>> *Date: *Sunday, October 6, 2024 at 06:22 >>>>>>>>>>>>>> *To: *Ángel <angel.alvarez.pas...@gmail.com> >>>>>>>>>>>>>> *Cc: *Russell Jurney <russell.jur...@gmail.com>, Mich >>>>>>>>>>>>>> Talebzadeh <mich.talebza...@gmail.com>, Spark dev list < >>>>>>>>>>>>>> dev@spark.apache.org>, user @spark <u...@spark.apache.org> >>>>>>>>>>>>>> *Subject: *Re: [DISCUSS] Deprecate GraphX OR Find new >>>>>>>>>>>>>> maintainers interested in GraphX OR leave it as is? >>>>>>>>>>>>>> >>>>>>>>>>>>>> So are there companies using it? And are they willing to >>>>>>>>>>>>>> contribute to maintaining it? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>> >>>>>>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f%3fq%3dhk_email&c=E,1,OT9ylxCx5xRNCToPSzu0VEvefs4uts16fTBydH2NiLHMGEwLjrEXgkhU8W-Ai6xD8VDMyWea44GBMOEecMNdapaZKZbBTrZpquOBKi6YRlqu-FVAzji6-w,,&typo=1> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,h0ccgHctUPRY4zAN_qZ-qdBgLDpQLtm7KaOL4u12U4PR7PeJ4MUBOS8bbD7CNssUIMqRMvY_pOqbh7PfLY0lRpQh9mfqBC0KnSHBZzxxSJJr-55r5kv6YjYwrA,,&typo=1> >>>>>>>>>>>>>> >>>>>>>>>>>>>> YouTube Live Streams: >>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>> >>>>>>>>>>>>>> Pronouns: she/her >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 9:17 PM Ángel < >>>>>>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> That would definitely affect companies using GraphX, but at >>>>>>>>>>>>>> least they’d have the choice to migrate their code. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think that’s probably the way to go. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> El dom, 6 oct 2024 a las 6:09, Holden Karau (< >>>>>>>>>>>>>> holden.ka...@gmail.com>) escribió: >>>>>>>>>>>>>> >>>>>>>>>>>>>> So removing GraphX from Spark would not prevent GraphFrames >>>>>>>>>>>>>> from continuing, they could pick up the GraphX source and >>>>>>>>>>>>>> incorporate it >>>>>>>>>>>>>> into their project. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>> >>>>>>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f%3fq%3dhk_email&c=E,1,9xMMQlY7gtmkqxT0NTmS8KMg4wOUjw0PWKM-oepAYAkE-SiM5pyXCb80AuRZYJ4zMIedVlwVMAKi_eh52Hof0LsteXx2eIslnsDBdmVeuocpILpneg,,&typo=1> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,kbGbMBRMidAYi0aqUmj949vRahpEjVzSgJv_YYtO5EteSXZy4RrMYXJU48mN2CyS5sdovsgiFAAiBLnyQ29gCCn8xbTrEJmfIhjtH7tD4N31VUoLtQ,,&typo=1> >>>>>>>>>>>>>> >>>>>>>>>>>>>> YouTube Live Streams: >>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>> >>>>>>>>>>>>>> Pronouns: she/her >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 5:22 PM Russell Jurney < >>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> A lot of people like me use GraphFrames for its connected >>>>>>>>>>>>>> components implementation and its motif matching feature. I am >>>>>>>>>>>>>> willing to >>>>>>>>>>>>>> work on it to keep it alive. They did a 0.8.3 release not too >>>>>>>>>>>>>> long ago. >>>>>>>>>>>>>> Please keep GraphX alive. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 3:44 PM Mich Talebzadeh < >>>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I added the user list as they may have vested interest here >>>>>>>>>>>>>> and and hopefully can contribute >>>>>>>>>>>>>> >>>>>>>>>>>>>> Few suggestions: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Data-Driven Decision Making: Return to the core >>>>>>>>>>>>>> metrics—analyze usage trends, performance benchmarks, and the >>>>>>>>>>>>>> actual impact >>>>>>>>>>>>>> on businesses that rely on GraphX. Objectivity can be >>>>>>>>>>>>>> restored by letting >>>>>>>>>>>>>> data speak louder than opinions so to speak. >>>>>>>>>>>>>> 2. Broaden the Discussion: Engage more stakeholders from >>>>>>>>>>>>>> diverse backgrounds (especially spark users) to bring in new >>>>>>>>>>>>>> perspectives >>>>>>>>>>>>>> and counterbalance the more vocal but potentially narrow >>>>>>>>>>>>>> interests of core >>>>>>>>>>>>>> maintainers or open-source contributors. >>>>>>>>>>>>>> 3. Define Clear Criteria for Decision Making: Agree on a >>>>>>>>>>>>>> set of objective criteria by which the project’s future will >>>>>>>>>>>>>> be judged. >>>>>>>>>>>>>> These could include market demand, contribution levels, >>>>>>>>>>>>>> maintenance costs, >>>>>>>>>>>>>> alternative solutions, and alignment with the overall Spark >>>>>>>>>>>>>> ecosystem >>>>>>>>>>>>>> goals. Some have already been covered. >>>>>>>>>>>>>> 4. Timely Conclusion of Discussions: Set a timeline for >>>>>>>>>>>>>> making a decision. Long, open-ended discussions tend to lose >>>>>>>>>>>>>> focus. Putting >>>>>>>>>>>>>> deadlines forces participants to focus on key issues and >>>>>>>>>>>>>> prevents endless >>>>>>>>>>>>>> debates. >>>>>>>>>>>>>> 5. Borrowing from commercial settings, it is often >>>>>>>>>>>>>> necessary for a strong leadership team to step in and make >>>>>>>>>>>>>> the final >>>>>>>>>>>>>> decision after considering the input. When the objectivity of >>>>>>>>>>>>>> discussions >>>>>>>>>>>>>> starts to wane, leadership needs to cut through the round >>>>>>>>>>>>>> discussions and >>>>>>>>>>>>>> steer towards action based on business and technical >>>>>>>>>>>>>> realities. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> HTH >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>>>>>>> >>>>>>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>>>>>>> College London >>>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>>>>>>> >>>>>>>>>>>>>> London, United Kingdom >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [image: Image removed by sender.] view my Linkedin profile >>>>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fen.everybodywiki.com%2fMich_Talebzadeh&c=E,1,U1JaGVMkko53HkJO5fwmkIXfziTOWL3K1CkAeHwFG55TbZQUd5xVNLGpLt2o0ytujE6zaLpqU2GWCZqHSbo3SU4Wh9Rl8NG4bWPbFWUwyw,,&typo=1> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>>>> essential >>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, 5 Oct 2024 at 06:26, Ángel < >>>>>>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I completely agree with everyone here. I don’t think the >>>>>>>>>>>>>> issue is deprecating it; to me, the problem lies in not >>>>>>>>>>>>>> providing a new and >>>>>>>>>>>>>> better solution for handling graphs in Spark. In the past, I >>>>>>>>>>>>>> used GraphX >>>>>>>>>>>>>> via GraphFrames for record linkage, and I found it both useful >>>>>>>>>>>>>> and >>>>>>>>>>>>>> effective. Is there any discussion about a potential replacement? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I’d be willing to help maintain GraphX, though I don’t have >>>>>>>>>>>>>> previous experience with maintaining open-source projects. All I >>>>>>>>>>>>>> can >>>>>>>>>>>>>> promise is good intentions, willingness to learn and lots of >>>>>>>>>>>>>> energy and >>>>>>>>>>>>>> passion. Is that enough? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Btw, what's your take on this? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> · *GraphX* will be deprecated in favor of a new >>>>>>>>>>>>>> graphing component, SparkGraph, based on Cypher >>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fneo4j.com%2fdeveloper%2fcypher-query-language%2f&c=E,1,5sP_K0oxQDLYIfWhFPwgNEmTuXMR7tvCjLLcf_ZBAv7oIBySxARy9TyrqNkmZKfXwrIDrhe6TVBCUun2luRV_mAbSD4rooD9YRt5GYYgbHbBUYerg1mpA4Oe6eo,&typo=1>, >>>>>>>>>>>>>> a much richer graph language than previously offered by GraphX. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://cloud.google.com/blog/products/data-analytics/introducing-spark-3-and-hadoop-3-on-dataproc-image-version-2-0 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> El sáb, 5 oct 2024 a las 2:17, Mark Hamstra (< >>>>>>>>>>>>>> markhams...@gmail.com>) escribió: >>>>>>>>>>>>>> >>>>>>>>>>>>>> As I wrote to Holden privately, I might well change my vote >>>>>>>>>>>>>> to be in >>>>>>>>>>>>>> favor of a deprecation label combined with some effective >>>>>>>>>>>>>> means of >>>>>>>>>>>>>> communicating that this doesn't mean the end for GraphX if >>>>>>>>>>>>>> interested >>>>>>>>>>>>>> contributors come forward to rescue it. I don't like either >>>>>>>>>>>>>> the idea >>>>>>>>>>>>>> of keeping unmaintained code and public APIs around >>>>>>>>>>>>>> (especially if >>>>>>>>>>>>>> there are problems with them) or the idea of removing Spark >>>>>>>>>>>>>> functionality just because no one has contributed to it for a >>>>>>>>>>>>>> while. A >>>>>>>>>>>>>> naked deprecation label feels somewhat drastic and >>>>>>>>>>>>>> pre-emptive to me. >>>>>>>>>>>>>> I don't expect that GraphX will be the last part of Spark to >>>>>>>>>>>>>> run the >>>>>>>>>>>>>> risk of death through neglect, and I think we need an >>>>>>>>>>>>>> effective means >>>>>>>>>>>>>> of encouraging resuscitation that a deprecation label on its >>>>>>>>>>>>>> own does >>>>>>>>>>>>>> not provide. On the other hand, if no one really is willing >>>>>>>>>>>>>> to come to >>>>>>>>>>>>>> the aid of GraphX or other neglected functionality given >>>>>>>>>>>>>> adequate >>>>>>>>>>>>>> warning of possible removal, I'm not then opposed to the usual >>>>>>>>>>>>>> deprecation and removal process. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Oct 4, 2024 at 4:10 PM Sean Owen <sro...@gmail.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > This is a reasonable discussion, but maybe the more >>>>>>>>>>>>>> practical point is: are you sure you want to block this >>>>>>>>>>>>>> unilaterally? This >>>>>>>>>>>>>> effectively makes a decision that GraphX cannot be removed for a >>>>>>>>>>>>>> long >>>>>>>>>>>>>> while. I'd understand it more if we had an active maintainer >>>>>>>>>>>>>> and/or active >>>>>>>>>>>>>> user proposing to veto, but my understanding is this is just a >>>>>>>>>>>>>> proposal to >>>>>>>>>>>>>> block this on behalf of some users, someone else who might do >>>>>>>>>>>>>> some work and >>>>>>>>>>>>>> hasn't to date for some reason. Add to that the fact that the >>>>>>>>>>>>>> 'pro' >>>>>>>>>>>>>> arguments all seem to be arguments for working on GraphFrames, >>>>>>>>>>>>>> and I find >>>>>>>>>>>>>> this somewhat drastic. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On Fri, Oct 4, 2024 at 5:23 PM Mark Hamstra < >>>>>>>>>>>>>> markhams...@gmail.com> wrote: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> "You can't say nothing is removable until there are no >>>>>>>>>>>>>> users." >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> That is not what I am saying. Rather, I am countering what >>>>>>>>>>>>>> others seem >>>>>>>>>>>>>> >> to be suggesting: There are no users and no interest, >>>>>>>>>>>>>> therefore we can >>>>>>>>>>>>>> >> and should deprecate. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> On Fri, Oct 4, 2024 at 3:10 PM Sean Owen <sro...@gmail.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > I could flip this argument around. More strongly, not >>>>>>>>>>>>>> being deprecated means "won't be removed" and likewise implies >>>>>>>>>>>>>> support and >>>>>>>>>>>>>> development. I don't think either of the latter have been true >>>>>>>>>>>>>> for years. >>>>>>>>>>>>>> What suggests this will change? A todo list is not going to do >>>>>>>>>>>>>> anything, >>>>>>>>>>>>>> IMHO. >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > I'm also concerned about the cost of that, which I have >>>>>>>>>>>>>> observed. GraphX PRs are almost certainly not going to be >>>>>>>>>>>>>> reviewed because >>>>>>>>>>>>>> of its state. Deprecation both communicates that reality, and >>>>>>>>>>>>>> leaves an >>>>>>>>>>>>>> option open, whereas not deprecating forecloses that option for >>>>>>>>>>>>>> a while. >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > I don't think the question is, does anyone use it? >>>>>>>>>>>>>> because anyone can continue to use it -- in Spark 3.x for sure, >>>>>>>>>>>>>> and in 4.x >>>>>>>>>>>>>> if not removed. >>>>>>>>>>>>>> >> > You can't say nothing is removable until there are no >>>>>>>>>>>>>> users. >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > Also, why would GraphFrames not be the logical home of >>>>>>>>>>>>>> this going forward anyway? which I think is the subtext. >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > On Fri, Oct 4, 2024 at 4:56 PM Mark Hamstra < >>>>>>>>>>>>>> markhams...@gmail.com> wrote: >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> I'm -1(*) because, while it technically means "might be >>>>>>>>>>>>>> removed in the >>>>>>>>>>>>>> >> >> future", I think developers and users are more prone to >>>>>>>>>>>>>> interpret >>>>>>>>>>>>>> >> >> something being marked as deprecated as "very likely >>>>>>>>>>>>>> will be removed >>>>>>>>>>>>>> >> >> in the future, so don't depend on this or waste your >>>>>>>>>>>>>> time contributing >>>>>>>>>>>>>> >> >> to its further development." I don't think the latter >>>>>>>>>>>>>> is what we want >>>>>>>>>>>>>> >> >> just because something hasn't been updated meaningfully >>>>>>>>>>>>>> in a while. >>>>>>>>>>>>>> >> >> There have been How To articles for GraphX and Graph >>>>>>>>>>>>>> Frames posted in >>>>>>>>>>>>>> >> >> the not too distant past, and the Google Search trend >>>>>>>>>>>>>> shows a pretty >>>>>>>>>>>>>> >> >> steady level of interest, not a decline to zero, so I >>>>>>>>>>>>>> don't think that >>>>>>>>>>>>>> >> >> it is accurate to declare that there is no use or >>>>>>>>>>>>>> interest in GraphX. >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> Unless retaining GraphX is imposing significant costs >>>>>>>>>>>>>> on continuing >>>>>>>>>>>>>> >> >> Spark development, I can't support deprecating GraphX. >>>>>>>>>>>>>> I can support >>>>>>>>>>>>>> >> >> encouraging GraphX and Graph Frames development through >>>>>>>>>>>>>> something like >>>>>>>>>>>>>> >> >> a To Do list or document of "What we'd like to see in >>>>>>>>>>>>>> the way of >>>>>>>>>>>>>> >> >> further development of Spark's graph processing >>>>>>>>>>>>>> capabilities" -- i.e., >>>>>>>>>>>>>> >> >> things that encourage and support new contributions to >>>>>>>>>>>>>> address any >>>>>>>>>>>>>> >> >> shortcomings in Spark's graph processing, not things >>>>>>>>>>>>>> that discourage >>>>>>>>>>>>>> >> >> contributions and use in the way that I believe simply >>>>>>>>>>>>>> declaring >>>>>>>>>>>>>> >> >> GraphX to be deprecated would. >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> On Sun, Sep 29, 2024 at 11:04 AM Holden Karau < >>>>>>>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>> >> >> > Since we're getting close to cutting a 4.0 branch I'd >>>>>>>>>>>>>> like to float the idea of officially deprecating Graph X. What >>>>>>>>>>>>>> that would >>>>>>>>>>>>>> mean (to me) is we would update the docs to indicate that Graph >>>>>>>>>>>>>> X is >>>>>>>>>>>>>> deprecated and it's APIs may be removed at anytime in the future. >>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>> >> >> > Alternatively, we could mark it as "unmaintained and >>>>>>>>>>>>>> in search of maintainers" with a note that if no maintainers are >>>>>>>>>>>>>> found, we >>>>>>>>>>>>>> may remove it in a future minor version. >>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>> >> >> > Looking at the source graph X, I don't see any >>>>>>>>>>>>>> meaningful active development going back over three years*. >>>>>>>>>>>>>> There is even a >>>>>>>>>>>>>> thread on user@ from 2017 asking if graph X is maintained >>>>>>>>>>>>>> anymore, with no response from the developers. >>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>> >> >> > Now I'm open to the idea that GraphX is stable and >>>>>>>>>>>>>> "works as is" and simply doesn't require modifications but given >>>>>>>>>>>>>> the user >>>>>>>>>>>>>> thread I'm a little concerned here about bringing this API with >>>>>>>>>>>>>> us into >>>>>>>>>>>>>> Spark 4 if we don't have anyone signed up to maintain it. >>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>> >> >> > * Excluding globally applied changes >>>>>>>>>>>>>> >> >> > -- >>>>>>>>>>>>>> >> >> > Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>> >> >> > Fight Health Insurance: >>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ >>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f&c=E,1,9CeJ-bKUShnxOFZMc15zJG1qgfAB9rnSDzrmLzNiXb8qE0NXedNCoZy4HobcS7laOMqtvJzYjvDzjBld1FaCPZpOBW6cf1l_xaG4bEbjYoDpNG0zuQ9_K5TW&typo=1> >>>>>>>>>>>>>> >> >> > Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,HJPBNbN3nfUZcb0-2OgveqIE5I5lvPSv-bOfRXIprFdSsGMlNq15o6rueLf2ZQRfytMu0-t3IxSjYou2uuPzUrSAqJ0LV42n2hG8rnkkpN4AA5w4mQZFTs4,&typo=1> >>>>>>>>>>>>>> >> >> > YouTube Live Streams: >>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>> >> >> > Pronouns: she/her >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>> >> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>>>>>> >>>>>>>>>>>>>>
