Hi Russell I have made the change suggested in jira and was able to run the tests after building. Opened up a PR <https://github.com/apache/spark/pull/48871>. Can you review it?
Regards Awadhesh On Mon, Nov 18, 2024 at 1:57 PM Russell Jurney <russell.jur...@gmail.com> wrote: > Thanks, I'm working on SPARK-42856 but the tests fail due to formatting > issues - confusing as I ran scalafmt. Working on it... > > Russ > > On Sun, Nov 17, 2024 at 7:05 PM Xiao Li <lix...@databricks.com> wrote: > >> Hi, Russell, >> >> >> After reviewing the JIRAs, it seems that only SPARK-42856 is directly >> relevant to GraphX. While the other three JIRAs mention GraphX in their >> descriptions, they appear to be more related to the build or the REPL >> rather than GraphX itself. >> >> Thanks, >> >> Xiao >> >> >> >> >> >> >> On Nov 16, 2024 at 5:39:27 PM, Russell Jurney <russell.jur...@gmail.com> >> wrote: >> >>> Scratch that, there appear to be... 4 unfixed bugs for GraphX >>> outstanding? :) >>> https://issues.apache.org/jira/browse/SPARK-42856?jql=project%20%3D%20SPARK%20AND%20issuetype%20%3D%20Bug%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22graphx%22 >>> >>> On Sat, Nov 16, 2024 at 5:23 PM Russell Jurney <russell.jur...@gmail.com> >>> wrote: >>> >>>> I'm looking at Spark's JIRA on a search for GraphX and I thought I >>>> would ask rather than just slog through it: anyone got some low hanging >>>> fruit bugs they can suggest I fix? >>>> >>>> Thanks, >>>> Russell >>>> >>>> On Thu, Nov 14, 2024 at 11:49 AM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> + 1 >>>>> >>>>> Mich Talebzadeh, >>>>> >>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>> College London <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>> >>>>> London, United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> >>>>> >>>>> >>>>> *Disclaimer:* The information provided is correct to the best of my >>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>> expert opinions (Werner >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>> >>>>> >>>>> On Thu, 14 Nov 2024 at 18:52, Russell Jurney <russell.jur...@gmail.com> >>>>> wrote: >>>>> >>>>>> Okay, first I’m going to fix a bug or two, I’ll get started on an >>>>>> SPIP. >>>>>> >>>>>> Russ >>>>>> >>>>>> On Wed, Nov 13, 2024 at 1:56 PM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> Hm. Since it sounds like a plan why Russell you go ahead and >>>>>>> create a SPIP for it, then, this discussion takes a formal approach and >>>>>>> is >>>>>>> documented. Otherwise we are just flogging a dead horse so to speak. >>>>>>> >>>>>>> HTH >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> >>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>> College London >>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>> London, United Kingdom >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>> >>>>>>> >>>>>>> On Wed, 13 Nov 2024 at 20:10, Russell Jurney < >>>>>>> russell.jur...@gmail.com> wrote: >>>>>>> >>>>>>>> It might be, but graph processing is a desirable, very useful >>>>>>>> feature of Spark. GraphX doesn't see more popularity because it never >>>>>>>> got a >>>>>>>> DataFrame interface. If someone is willing to add one and maintain it, >>>>>>>> that >>>>>>>> seems best of all. >>>>>>>> >>>>>>>> Russ >>>>>>>> >>>>>>>> On Wed, Nov 13, 2024 at 7:12 AM Ángel < >>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Seems to me.... it would be easier to move GraphX to graphframes >>>>>>>>> than the opposite. >>>>>>>>> >>>>>>>>> El mar, 8 oct 2024 a las 21:52, Reynold Xin >>>>>>>>> (<r...@databricks.com.invalid>) escribió: >>>>>>>>> >>>>>>>>>> We can also consider the following: move GraphFrame into Spark, >>>>>>>>>> and make GraphX an internal impl detail of GraphFrame. Then we can >>>>>>>>>> over >>>>>>>>>> time change the implementation, simplify it (not sure if it is >>>>>>>>>> possible, >>>>>>>>>> but somebody can look into it).... >>>>>>>>>> >>>>>>>>>> On Mon, Oct 7, 2024 at 7:04 PM Russell Jurney < >>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Took a look at recent activity. Spark 3.5 support >>>>>>>>>>> <https://github.com/graphframes/graphframes/commit/e54f249605dde60787f9b41b88ed7d5872b7dfab> >>>>>>>>>>> was >>>>>>>>>>> added a year ago. I'm sure we'll add Spark 4 support as soon as it >>>>>>>>>>> is out. >>>>>>>>>>> >>>>>>>>>>> There is a new issue to organize a GraphFrames Hackathon >>>>>>>>>>> <https://github.com/graphframes/graphframes/issues/460>. Please >>>>>>>>>>> sign up to help! >>>>>>>>>>> https://github.com/graphframes/graphframes/issues/460 >>>>>>>>>>> >>>>>>>>>>> I seriously need GraphX and GraphFrames to make it... I have no >>>>>>>>>>> other way of doing property graph motif matching on large graphs. >>>>>>>>>>> It's kind >>>>>>>>>>> of important to me. >>>>>>>>>>> >>>>>>>>>>> Some slides on my work with GraphFrames: >>>>>>>>>>> >>>>>>>>>>> [image: image.png] >>>>>>>>>>> >>>>>>>>>>> [image: image.png] >>>>>>>>>>> >>>>>>>>>>> [image: image.png] >>>>>>>>>>> >>>>>>>>>>> [image: image.png] >>>>>>>>>>> >>>>>>>>>>> [image: image.png] >>>>>>>>>>> >>>>>>>>>>> Russell >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Oct 7, 2024 at 6:06 PM Holden Karau < >>>>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> That’s awesome! >>>>>>>>>>>> >>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>>>> Pronouns: she/her >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Oct 7, 2024 at 5:42 PM Russell Jurney < >>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I’ll organize a hackathon. A friend wants to finish the >>>>>>>>>>>>> implementation of Lucian modularity for GraphFrames. I’ll fix >>>>>>>>>>>>> some GraphX >>>>>>>>>>>>> bugs at it. >>>>>>>>>>>>> >>>>>>>>>>>>> I did just blog all about the motif matching in GraphFrames: >>>>>>>>>>>>> >>>>>>>>>>>>> https://blog.graphlet.ai/financial-crime-and-corruption-network-motifs-4cf2e8e10eb5 >>>>>>>>>>>>> >>>>>>>>>>>>> Russ >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Oct 7, 2024 at 5:38 PM Holden Karau < >>>>>>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> So this discuss thread and the vote thread to deprecate to >>>>>>>>>>>>>> leave the option of removing it during 4.X are probably the >>>>>>>>>>>>>> highest profile >>>>>>>>>>>>>> it’s been in years. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the past for parts of Spark I’ve cared about I’ve >>>>>>>>>>>>>> organized virtual meetings to co-ordinate work — if your >>>>>>>>>>>>>> connected with >>>>>>>>>>>>>> some of the Spark+Graph community reaching out to find others and >>>>>>>>>>>>>> organizing a meeting could be a way to raise the profile a bit? >>>>>>>>>>>>>> Maybe >>>>>>>>>>>>>> organize a virtual hackathon (I’m meaning to try this for some >>>>>>>>>>>>>> other things >>>>>>>>>>>>>> so happy to share what I learn from doing that)? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>>>>>> YouTube Live Streams: >>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>> Pronouns: she/her >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 5:02 PM Russell Jurney < >>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I’ll look for a bug to fix. If GraphX is outside of Spark, >>>>>>>>>>>>>>> Spark would tend to break GraphFrames and it will be burdensome >>>>>>>>>>>>>>> on an >>>>>>>>>>>>>>> external project to keep up. Graph computing on Spark is >>>>>>>>>>>>>>> implrtant to a lot >>>>>>>>>>>>>>> of people, is there a way to raise visibility here? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 4:24 PM Holden Karau < >>>>>>>>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There are no specific tickets associated with the lack of >>>>>>>>>>>>>>>> maintaince or this as the component has not been maintained >>>>>>>>>>>>>>>> for a >>>>>>>>>>>>>>>> sufficiently long time. If your interested in taking it on >>>>>>>>>>>>>>>> that’s >>>>>>>>>>>>>>>> wonderful, probably starting with fixing some bugs could be a >>>>>>>>>>>>>>>> great place >>>>>>>>>>>>>>>> to start and figure out if it’s something you want to do long >>>>>>>>>>>>>>>> term. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would recommend making a first bug fix in a actively >>>>>>>>>>>>>>>> maintained area of Spark to get to >>>>>>>>>>>>>>>> Know some reviewers since there is not anyone tracking the >>>>>>>>>>>>>>>> GraphX PRs. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As a note I don’t think GraphX is required for Graph Frames >>>>>>>>>>>>>>>> long term, so another option would be to talk to the >>>>>>>>>>>>>>>> GraphFrames folks and >>>>>>>>>>>>>>>> move the GraphX code over to it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ideally we’d have someone willing to act as a mentor or >>>>>>>>>>>>>>>> guide but so far we have no volunteers (especially no one >>>>>>>>>>>>>>>> familiar with the >>>>>>>>>>>>>>>> graph X code). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>>>> Fight Health Insurance: >>>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ >>>>>>>>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>>>>>>>> YouTube Live Streams: >>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>>>> Pronouns: she/her >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 3:25 PM Russell Jurney < >>>>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I volunteer to maintain GraphX to keep GraphFrames a >>>>>>>>>>>>>>>>> viable project. I don’t have a clear view on whether it works >>>>>>>>>>>>>>>>> with Spark 4 >>>>>>>>>>>>>>>>> or if it needs updates? I don’t have Spark commits but I’m a >>>>>>>>>>>>>>>>> committer on >>>>>>>>>>>>>>>>> Apache DataFu and mentored the Spark feature for it. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can someone tell me what is involved? Point me at a ticket? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Russell >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 12:11 AM Erik Eklund < >>>>>>>>>>>>>>>>> eekl...@definitivehc.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>> We rely on GraphX for an important component of our >>>>>>>>>>>>>>>>>> product. And we really want it to stay a typed interface. >>>>>>>>>>>>>>>>>> Please keep >>>>>>>>>>>>>>>>>> GraphX. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Erik >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *From: *Holden Karau <holden.ka...@gmail.com> >>>>>>>>>>>>>>>>>> *Date: *Sunday, October 6, 2024 at 06:22 >>>>>>>>>>>>>>>>>> *To: *Ángel <angel.alvarez.pas...@gmail.com> >>>>>>>>>>>>>>>>>> *Cc: *Russell Jurney <russell.jur...@gmail.com>, Mich >>>>>>>>>>>>>>>>>> Talebzadeh <mich.talebza...@gmail.com>, Spark dev list < >>>>>>>>>>>>>>>>>> dev@spark.apache.org>, user @spark <u...@spark.apache.org >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> *Subject: *Re: [DISCUSS] Deprecate GraphX OR Find new >>>>>>>>>>>>>>>>>> maintainers interested in GraphX OR leave it as is? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So are there companies using it? And are they willing to >>>>>>>>>>>>>>>>>> contribute to maintaining it? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Fight Health Insurance: >>>>>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ >>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f%3fq%3dhk_email&c=E,1,OT9ylxCx5xRNCToPSzu0VEvefs4uts16fTBydH2NiLHMGEwLjrEXgkhU8W-Ai6xD8VDMyWea44GBMOEecMNdapaZKZbBTrZpquOBKi6YRlqu-FVAzji6-w,,&typo=1> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,h0ccgHctUPRY4zAN_qZ-qdBgLDpQLtm7KaOL4u12U4PR7PeJ4MUBOS8bbD7CNssUIMqRMvY_pOqbh7PfLY0lRpQh9mfqBC0KnSHBZzxxSJJr-55r5kv6YjYwrA,,&typo=1> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> YouTube Live Streams: >>>>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Pronouns: she/her >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 9:17 PM Ángel < >>>>>>>>>>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> That would definitely affect companies using GraphX, but >>>>>>>>>>>>>>>>>> at least they’d have the choice to migrate their code. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I think that’s probably the way to go. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> El dom, 6 oct 2024 a las 6:09, Holden Karau (< >>>>>>>>>>>>>>>>>> holden.ka...@gmail.com>) escribió: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So removing GraphX from Spark would not prevent >>>>>>>>>>>>>>>>>> GraphFrames from continuing, they could pick up the GraphX >>>>>>>>>>>>>>>>>> source and >>>>>>>>>>>>>>>>>> incorporate it into their project. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Fight Health Insurance: >>>>>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ >>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f%3fq%3dhk_email&c=E,1,9xMMQlY7gtmkqxT0NTmS8KMg4wOUjw0PWKM-oepAYAkE-SiM5pyXCb80AuRZYJ4zMIedVlwVMAKi_eh52Hof0LsteXx2eIslnsDBdmVeuocpILpneg,,&typo=1> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,kbGbMBRMidAYi0aqUmj949vRahpEjVzSgJv_YYtO5EteSXZy4RrMYXJU48mN2CyS5sdovsgiFAAiBLnyQ29gCCn8xbTrEJmfIhjtH7tD4N31VUoLtQ,,&typo=1> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> YouTube Live Streams: >>>>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Pronouns: she/her >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 5:22 PM Russell Jurney < >>>>>>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> A lot of people like me use GraphFrames for its connected >>>>>>>>>>>>>>>>>> components implementation and its motif matching feature. I >>>>>>>>>>>>>>>>>> am willing to >>>>>>>>>>>>>>>>>> work on it to keep it alive. They did a 0.8.3 release not >>>>>>>>>>>>>>>>>> too long ago. >>>>>>>>>>>>>>>>>> Please keep GraphX alive. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 3:44 PM Mich Talebzadeh < >>>>>>>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I added the user list as they may have vested >>>>>>>>>>>>>>>>>> interest here and and hopefully can contribute >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Few suggestions: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. Data-Driven Decision Making: Return to the core >>>>>>>>>>>>>>>>>> metrics—analyze usage trends, performance benchmarks, and >>>>>>>>>>>>>>>>>> the actual impact >>>>>>>>>>>>>>>>>> on businesses that rely on GraphX. Objectivity can be >>>>>>>>>>>>>>>>>> restored by letting >>>>>>>>>>>>>>>>>> data speak louder than opinions so to speak. >>>>>>>>>>>>>>>>>> 2. Broaden the Discussion: Engage more stakeholders >>>>>>>>>>>>>>>>>> from diverse backgrounds (especially spark users) to >>>>>>>>>>>>>>>>>> bring in new >>>>>>>>>>>>>>>>>> perspectives and counterbalance the more vocal but >>>>>>>>>>>>>>>>>> potentially narrow >>>>>>>>>>>>>>>>>> interests of core maintainers or open-source contributors. >>>>>>>>>>>>>>>>>> 3. Define Clear Criteria for Decision Making: Agree >>>>>>>>>>>>>>>>>> on a set of objective criteria by which the project’s >>>>>>>>>>>>>>>>>> future will be >>>>>>>>>>>>>>>>>> judged. These could include market demand, contribution >>>>>>>>>>>>>>>>>> levels, maintenance >>>>>>>>>>>>>>>>>> costs, alternative solutions, and alignment with the >>>>>>>>>>>>>>>>>> overall Spark >>>>>>>>>>>>>>>>>> ecosystem goals. Some have already been covered. >>>>>>>>>>>>>>>>>> 4. Timely Conclusion of Discussions: Set a timeline >>>>>>>>>>>>>>>>>> for making a decision. Long, open-ended discussions tend >>>>>>>>>>>>>>>>>> to lose focus. >>>>>>>>>>>>>>>>>> Putting deadlines forces participants to focus on key >>>>>>>>>>>>>>>>>> issues and prevents >>>>>>>>>>>>>>>>>> endless debates. >>>>>>>>>>>>>>>>>> 5. Borrowing from commercial settings, it is often >>>>>>>>>>>>>>>>>> necessary for a strong leadership team to step in and >>>>>>>>>>>>>>>>>> make the final >>>>>>>>>>>>>>>>>> decision after considering the input. When the >>>>>>>>>>>>>>>>>> objectivity of discussions >>>>>>>>>>>>>>>>>> starts to wane, leadership needs to cut through the round >>>>>>>>>>>>>>>>>> discussions and >>>>>>>>>>>>>>>>>> steer towards action based on business and technical >>>>>>>>>>>>>>>>>> realities. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> HTH >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> >>>>>>>>>>>>>>>>>> Imperial >>>>>>>>>>>>>>>>>> College London >>>>>>>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> London, United Kingdom >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [image: Image removed by sender.] view my Linkedin >>>>>>>>>>>>>>>>>> profile >>>>>>>>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fen.everybodywiki.com%2fMich_Talebzadeh&c=E,1,U1JaGVMkko53HkJO5fwmkIXfziTOWL3K1CkAeHwFG55TbZQUd5xVNLGpLt2o0ytujE6zaLpqU2GWCZqHSbo3SU4Wh9Rl8NG4bWPbFWUwyw,,&typo=1> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It >>>>>>>>>>>>>>>>>> is essential >>>>>>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von >>>>>>>>>>>>>>>>>> Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun> >>>>>>>>>>>>>>>>>> )". >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, 5 Oct 2024 at 06:26, Ángel < >>>>>>>>>>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I completely agree with everyone here. I don’t think the >>>>>>>>>>>>>>>>>> issue is deprecating it; to me, the problem lies in not >>>>>>>>>>>>>>>>>> providing a new and >>>>>>>>>>>>>>>>>> better solution for handling graphs in Spark. In the past, I >>>>>>>>>>>>>>>>>> used GraphX >>>>>>>>>>>>>>>>>> via GraphFrames for record linkage, and I found it both >>>>>>>>>>>>>>>>>> useful and >>>>>>>>>>>>>>>>>> effective. Is there any discussion about a potential >>>>>>>>>>>>>>>>>> replacement? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I’d be willing to help maintain GraphX, though I don’t >>>>>>>>>>>>>>>>>> have previous experience with maintaining open-source >>>>>>>>>>>>>>>>>> projects. All I can >>>>>>>>>>>>>>>>>> promise is good intentions, willingness to learn and lots of >>>>>>>>>>>>>>>>>> energy and >>>>>>>>>>>>>>>>>> passion. Is that enough? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Btw, what's your take on this? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> · *GraphX* will be deprecated in favor of a new >>>>>>>>>>>>>>>>>> graphing component, SparkGraph, based on Cypher >>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fneo4j.com%2fdeveloper%2fcypher-query-language%2f&c=E,1,5sP_K0oxQDLYIfWhFPwgNEmTuXMR7tvCjLLcf_ZBAv7oIBySxARy9TyrqNkmZKfXwrIDrhe6TVBCUun2luRV_mAbSD4rooD9YRt5GYYgbHbBUYerg1mpA4Oe6eo,&typo=1>, >>>>>>>>>>>>>>>>>> a much richer graph language than previously offered by >>>>>>>>>>>>>>>>>> GraphX. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://cloud.google.com/blog/products/data-analytics/introducing-spark-3-and-hadoop-3-on-dataproc-image-version-2-0 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> El sáb, 5 oct 2024 a las 2:17, Mark Hamstra (< >>>>>>>>>>>>>>>>>> markhams...@gmail.com>) escribió: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> As I wrote to Holden privately, I might well change my >>>>>>>>>>>>>>>>>> vote to be in >>>>>>>>>>>>>>>>>> favor of a deprecation label combined with some effective >>>>>>>>>>>>>>>>>> means of >>>>>>>>>>>>>>>>>> communicating that this doesn't mean the end for GraphX >>>>>>>>>>>>>>>>>> if interested >>>>>>>>>>>>>>>>>> contributors come forward to rescue it. I don't like >>>>>>>>>>>>>>>>>> either the idea >>>>>>>>>>>>>>>>>> of keeping unmaintained code and public APIs around >>>>>>>>>>>>>>>>>> (especially if >>>>>>>>>>>>>>>>>> there are problems with them) or the idea of removing >>>>>>>>>>>>>>>>>> Spark >>>>>>>>>>>>>>>>>> functionality just because no one has contributed to it >>>>>>>>>>>>>>>>>> for a while. A >>>>>>>>>>>>>>>>>> naked deprecation label feels somewhat drastic and >>>>>>>>>>>>>>>>>> pre-emptive to me. >>>>>>>>>>>>>>>>>> I don't expect that GraphX will be the last part of Spark >>>>>>>>>>>>>>>>>> to run the >>>>>>>>>>>>>>>>>> risk of death through neglect, and I think we need an >>>>>>>>>>>>>>>>>> effective means >>>>>>>>>>>>>>>>>> of encouraging resuscitation that a deprecation label on >>>>>>>>>>>>>>>>>> its own does >>>>>>>>>>>>>>>>>> not provide. On the other hand, if no one really is >>>>>>>>>>>>>>>>>> willing to come to >>>>>>>>>>>>>>>>>> the aid of GraphX or other neglected functionality given >>>>>>>>>>>>>>>>>> adequate >>>>>>>>>>>>>>>>>> warning of possible removal, I'm not then opposed to the >>>>>>>>>>>>>>>>>> usual >>>>>>>>>>>>>>>>>> deprecation and removal process. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Oct 4, 2024 at 4:10 PM Sean Owen < >>>>>>>>>>>>>>>>>> sro...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > This is a reasonable discussion, but maybe the more >>>>>>>>>>>>>>>>>> practical point is: are you sure you want to block this >>>>>>>>>>>>>>>>>> unilaterally? This >>>>>>>>>>>>>>>>>> effectively makes a decision that GraphX cannot be removed >>>>>>>>>>>>>>>>>> for a long >>>>>>>>>>>>>>>>>> while. I'd understand it more if we had an active maintainer >>>>>>>>>>>>>>>>>> and/or active >>>>>>>>>>>>>>>>>> user proposing to veto, but my understanding is this is just >>>>>>>>>>>>>>>>>> a proposal to >>>>>>>>>>>>>>>>>> block this on behalf of some users, someone else who might >>>>>>>>>>>>>>>>>> do some work and >>>>>>>>>>>>>>>>>> hasn't to date for some reason. Add to that the fact that >>>>>>>>>>>>>>>>>> the 'pro' >>>>>>>>>>>>>>>>>> arguments all seem to be arguments for working on >>>>>>>>>>>>>>>>>> GraphFrames, and I find >>>>>>>>>>>>>>>>>> this somewhat drastic. >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > On Fri, Oct 4, 2024 at 5:23 PM Mark Hamstra < >>>>>>>>>>>>>>>>>> markhams...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> "You can't say nothing is removable until there are no >>>>>>>>>>>>>>>>>> users." >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> That is not what I am saying. Rather, I am countering >>>>>>>>>>>>>>>>>> what others seem >>>>>>>>>>>>>>>>>> >> to be suggesting: There are no users and no interest, >>>>>>>>>>>>>>>>>> therefore we can >>>>>>>>>>>>>>>>>> >> and should deprecate. >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> On Fri, Oct 4, 2024 at 3:10 PM Sean Owen < >>>>>>>>>>>>>>>>>> sro...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>> >> > I could flip this argument around. More strongly, >>>>>>>>>>>>>>>>>> not being deprecated means "won't be removed" and likewise >>>>>>>>>>>>>>>>>> implies support >>>>>>>>>>>>>>>>>> and development. I don't think either of the latter have >>>>>>>>>>>>>>>>>> been true for >>>>>>>>>>>>>>>>>> years. What suggests this will change? A todo list is not >>>>>>>>>>>>>>>>>> going to do >>>>>>>>>>>>>>>>>> anything, IMHO. >>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>> >> > I'm also concerned about the cost of that, which I >>>>>>>>>>>>>>>>>> have observed. GraphX PRs are almost certainly not going to >>>>>>>>>>>>>>>>>> be reviewed >>>>>>>>>>>>>>>>>> because of its state. Deprecation both communicates that >>>>>>>>>>>>>>>>>> reality, and >>>>>>>>>>>>>>>>>> leaves an option open, whereas not deprecating forecloses >>>>>>>>>>>>>>>>>> that option for a >>>>>>>>>>>>>>>>>> while. >>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>> >> > I don't think the question is, does anyone use it? >>>>>>>>>>>>>>>>>> because anyone can continue to use it -- in Spark 3.x for >>>>>>>>>>>>>>>>>> sure, and in 4.x >>>>>>>>>>>>>>>>>> if not removed. >>>>>>>>>>>>>>>>>> >> > You can't say nothing is removable until there are >>>>>>>>>>>>>>>>>> no users. >>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>> >> > Also, why would GraphFrames not be the logical home >>>>>>>>>>>>>>>>>> of this going forward anyway? which I think is the subtext. >>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>> >> > On Fri, Oct 4, 2024 at 4:56 PM Mark Hamstra < >>>>>>>>>>>>>>>>>> markhams...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> I'm -1(*) because, while it technically means >>>>>>>>>>>>>>>>>> "might be removed in the >>>>>>>>>>>>>>>>>> >> >> future", I think developers and users are more >>>>>>>>>>>>>>>>>> prone to interpret >>>>>>>>>>>>>>>>>> >> >> something being marked as deprecated as "very >>>>>>>>>>>>>>>>>> likely will be removed >>>>>>>>>>>>>>>>>> >> >> in the future, so don't depend on this or waste >>>>>>>>>>>>>>>>>> your time contributing >>>>>>>>>>>>>>>>>> >> >> to its further development." I don't think the >>>>>>>>>>>>>>>>>> latter is what we want >>>>>>>>>>>>>>>>>> >> >> just because something hasn't been updated >>>>>>>>>>>>>>>>>> meaningfully in a while. >>>>>>>>>>>>>>>>>> >> >> There have been How To articles for GraphX and >>>>>>>>>>>>>>>>>> Graph Frames posted in >>>>>>>>>>>>>>>>>> >> >> the not too distant past, and the Google Search >>>>>>>>>>>>>>>>>> trend shows a pretty >>>>>>>>>>>>>>>>>> >> >> steady level of interest, not a decline to zero, so >>>>>>>>>>>>>>>>>> I don't think that >>>>>>>>>>>>>>>>>> >> >> it is accurate to declare that there is no use or >>>>>>>>>>>>>>>>>> interest in GraphX. >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> Unless retaining GraphX is imposing significant >>>>>>>>>>>>>>>>>> costs on continuing >>>>>>>>>>>>>>>>>> >> >> Spark development, I can't support deprecating >>>>>>>>>>>>>>>>>> GraphX. I can support >>>>>>>>>>>>>>>>>> >> >> encouraging GraphX and Graph Frames development >>>>>>>>>>>>>>>>>> through something like >>>>>>>>>>>>>>>>>> >> >> a To Do list or document of "What we'd like to see >>>>>>>>>>>>>>>>>> in the way of >>>>>>>>>>>>>>>>>> >> >> further development of Spark's graph processing >>>>>>>>>>>>>>>>>> capabilities" -- i.e., >>>>>>>>>>>>>>>>>> >> >> things that encourage and support new contributions >>>>>>>>>>>>>>>>>> to address any >>>>>>>>>>>>>>>>>> >> >> shortcomings in Spark's graph processing, not >>>>>>>>>>>>>>>>>> things that discourage >>>>>>>>>>>>>>>>>> >> >> contributions and use in the way that I believe >>>>>>>>>>>>>>>>>> simply declaring >>>>>>>>>>>>>>>>>> >> >> GraphX to be deprecated would. >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> On Sun, Sep 29, 2024 at 11:04 AM Holden Karau < >>>>>>>>>>>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>> >> >> > Since we're getting close to cutting a 4.0 branch >>>>>>>>>>>>>>>>>> I'd like to float the idea of officially deprecating Graph >>>>>>>>>>>>>>>>>> X. What that >>>>>>>>>>>>>>>>>> would mean (to me) is we would update the docs to indicate >>>>>>>>>>>>>>>>>> that Graph X is >>>>>>>>>>>>>>>>>> deprecated and it's APIs may be removed at anytime in the >>>>>>>>>>>>>>>>>> future. >>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>> >> >> > Alternatively, we could mark it as "unmaintained >>>>>>>>>>>>>>>>>> and in search of maintainers" with a note that if no >>>>>>>>>>>>>>>>>> maintainers are found, >>>>>>>>>>>>>>>>>> we may remove it in a future minor version. >>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>> >> >> > Looking at the source graph X, I don't see any >>>>>>>>>>>>>>>>>> meaningful active development going back over three years*. >>>>>>>>>>>>>>>>>> There is even a >>>>>>>>>>>>>>>>>> thread on user@ from 2017 asking if graph X is >>>>>>>>>>>>>>>>>> maintained anymore, with no response from the developers. >>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>> >> >> > Now I'm open to the idea that GraphX is stable >>>>>>>>>>>>>>>>>> and "works as is" and simply doesn't require modifications >>>>>>>>>>>>>>>>>> but given the >>>>>>>>>>>>>>>>>> user thread I'm a little concerned here about bringing this >>>>>>>>>>>>>>>>>> API with us >>>>>>>>>>>>>>>>>> into Spark 4 if we don't have anyone signed up to maintain >>>>>>>>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>> >> >> > * Excluding globally applied changes >>>>>>>>>>>>>>>>>> >> >> > -- >>>>>>>>>>>>>>>>>> >> >> > Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>>>>>> >> >> > Fight Health Insurance: >>>>>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ >>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f&c=E,1,9CeJ-bKUShnxOFZMc15zJG1qgfAB9rnSDzrmLzNiXb8qE0NXedNCoZy4HobcS7laOMqtvJzYjvDzjBld1FaCPZpOBW6cf1l_xaG4bEbjYoDpNG0zuQ9_K5TW&typo=1> >>>>>>>>>>>>>>>>>> >> >> > Books (Learning Spark, High Performance Spark, >>>>>>>>>>>>>>>>>> etc.): https://amzn.to/2MaRAG9 >>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,HJPBNbN3nfUZcb0-2OgveqIE5I5lvPSv-bOfRXIprFdSsGMlNq15o6rueLf2ZQRfytMu0-t3IxSjYou2uuPzUrSAqJ0LV42n2hG8rnkkpN4AA5w4mQZFTs4,&typo=1> >>>>>>>>>>>>>>>>>> >> >> > YouTube Live Streams: >>>>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>>>>>> >> >> > Pronouns: she/her >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>>>> >> >> To unsubscribe e-mail: >>>>>>>>>>>>>>>>>> dev-unsubscr...@spark.apache.org >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>
