👍 El mié, 13 nov 2024, 3:52, Holden Karau <holden.ka...@gmail.com> escribió:
> So it’s deprecated but I will review some basic graph X PRs as I would > like us to bring graph X back to life — but under our current release > structure we need to deprecate now if we want to be able to remove it in > the next few years. > > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her > > > On Tue, Nov 12, 2024 at 6:47 PM Ángel <angel.alvarez.pas...@gmail.com> > wrote: > >> I thought that too ... until I read the message from Matei Zaharia: >> >> "Votes to deprecate both SparkR and GraphX have passed. These components >> will officially be deprecated in Spark 4." >> >> Didn't know in open source you could deprecate things that have been >> there years so lightly without carrying out any impact analysis and in the >> middle of an active (and interesting, btw) discussion. >> >> El mar, 12 nov 2024, 21:59, Russell Jurney <russell.jur...@gmail.com> >> escribió: >> >>> That is unfortunate. I saw someone volunteer to review my PRs. I thought >>> there was a holdout? >>> >>> On Tue, Nov 12, 2024 at 12:56 PM Ángel <angel.alvarez.pas...@gmail.com> >>> wrote: >>> >>>> Nope. didn't miss that, in fact, I mentioned that graphframes used >>>> GraphX under the hood. >>>> >>>> The thing is ... even though we were trying to get maintainers the >>>> deprecation of GraphX passed suddenly in the middle of that discussion. >>>> >>>> El mar, 12 nov 2024, 21:47, Russell Jurney <russell.jur...@gmail.com> >>>> escribió: >>>> >>>>> I guess you missed where Reynold Xin suggested we instead bring >>>>> GraphFrames into Spark and others agreed? >>>>> >>>>> On Tue, Nov 12, 2024 at 12:08 PM Ángel <angel.alvarez.pas...@gmail.com> >>>>> wrote: >>>>> >>>>>> You only have to look at the subject of this thread of mails. It says >>>>>> nothing about graphframes. I thought we were "fighting" against >>>>>> deprecating >>>>>> GraphX because it seemed not have any maintainers in quite a few time. >>>>>> Maybe I got it wrong. >>>>>> >>>>>> El mar, 12 nov 2024, 19:12, Russell Jurney <russell.jur...@gmail.com> >>>>>> escribió: >>>>>> >>>>>>> Not sure what you mean? GraphX is the core Apache Spark technology >>>>>>> underneath GraphFrames - parts of GraphFrames use it. `git grep -i >>>>>>> graphx | >>>>>>> wc -l` shows 147 hits for `graphx` in GraphFrames master branch as of >>>>>>> now. >>>>>>> >>>>>>> I started out getting familiar with the GraphFrames codebase with >>>>>>> some low hanging fixes for housekeeping - GraphFrames hasn't been >>>>>>> maintained, the docs are outdated and most examples didn't work. The >>>>>>> friends graph didn't even load in all of the examples, as the Python >>>>>>> graphframes.examples.Graphs interface changed to be object oriented. I >>>>>>> am >>>>>>> now a maintainer of the Github repo thanks to Weichen Xu, who is >>>>>>> reviewing >>>>>>> my PRs along with Reynold Xin (very nice, thanks!). I am getting >>>>>>> familiar >>>>>>> with GraphFrames internals before looking at what it takes to get it >>>>>>> into >>>>>>> Apache Spark proper. I was just letting people know that I am actively >>>>>>> taking on the work that we are discussing. >>>>>>> >>>>>>> Next, I am also going to get to know GraphX in the next couple of >>>>>>> weeks to better understand what moving GraphFrames into Spark would look >>>>>>> like. >>>>>>> >>>>>>> In a week or so, I will announce the hackathon. Need to figure out a >>>>>>> time that works for the people who've already expressed interest. >>>>>>> >>>>>>> Thanks, >>>>>>> Russell Jurney >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Nov 12, 2024 at 6:48 AM Ángel < >>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>> >>>>>>>> But the goal wasn't to fix bugs in GraphX? What has that to do with >>>>>>>> graphframes? >>>>>>>> >>>>>>>> El mar, 12 nov 2024, 12:58, Russell Jurney < >>>>>>>> russell.jur...@gmail.com> escribió: >>>>>>>> >>>>>>>>> I started working on GraphFrames this weekend, got it building and >>>>>>>>> started with some docs PRs. A lot of the example code no longer >>>>>>>>> worked, so >>>>>>>>> I fixed it. I'm updating the docs to indicate our plan to integrate >>>>>>>>> it with >>>>>>>>> Apache Spark. >>>>>>>>> >>>>>>>>> I'll announce a hackathon in the next week or so :) >>>>>>>>> >>>>>>>>> Russell >>>>>>>>> >>>>>>>>> On Wed, Oct 16, 2024 at 6:53 PM Russell Jurney < >>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> For starters I created a ticket. I'm going to work on the project >>>>>>>>>> a bit and then name a date and time. >>>>>>>>>> >>>>>>>>>> https://github.com/graphframes/graphframes/issues/460 >>>>>>>>>> >>>>>>>>>> On Tue, Oct 15, 2024 at 7:48 PM Ángel < >>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> We could create a prioritized list of the most important bugs to >>>>>>>>>>> fix first and distribute the tasks among us. We can also share the >>>>>>>>>>> knowledge we gain from resolving them. >>>>>>>>>>> btw, what happened to the (great) hackathon idea? any date/s in >>>>>>>>>>> mind? >>>>>>>>>>> >>>>>>>>>>> El mié, 16 oct 2024 a las 3:53, Russell Jurney (< >>>>>>>>>>> russell.jur...@gmail.com>) escribió: >>>>>>>>>>> >>>>>>>>>>>> I've never used Visual Studio Code for Java / Scala, but I find >>>>>>>>>>>> IntelliJ works really well. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Oct 15, 2024 at 4:58 PM Mich Talebzadeh < >>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> VSC is also another good candidate heavily used in the banks. >>>>>>>>>>>>> Anyway you parted with money so all the best. >>>>>>>>>>>>> >>>>>>>>>>>>> cheers >>>>>>>>>>>>> >>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>> >>>>>>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>>>>>> College London >>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>>>>>> London, United Kingdom >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>>> essential to >>>>>>>>>>>>> note that, as with any advice, quote "one test result is >>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>
