A quick message from the ASF Infra team here regarding this thread. As of a couple of minutes ago, attached images are no longer supported on this mailing list in the foreseeable future. The TL;DR of that is that these 20 megabyte emails are sent out to several thousands of people, which in turn generates hundreds of gigabytes of temporary disk space usage while such an email is sent out. To add further insult to injury, GMail is refusing these emails because of their size and volume, meaning we have thousands upon thousands of 20MB emails clogging up our servers.
This is an unfortunate turn of events, we do not wish to block attachments, but right now we have no choice. We will let you know if anything changes in the near future. If you have any questions (other than "when can we have images back", we do not know), please feel free to reach out to us at us...@infra.apache.org With regards, Daniel on behalf of ASF Infra. On 2024/10/15 23:57:47 Mich Talebzadeh wrote: > VSC is also another good candidate heavily used in the banks. Anyway you > parted with money so all the best. > > cheers > > Mich Talebzadeh, > > Architect | Data Engineer | Data Science | Financial Crime > PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College > London <https://en.wikipedia.org/wiki/Imperial_College_London> > London, United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Wed, 16 Oct 2024 at 00:22, Russell Jurney <russell.jur...@gmail.com> > wrote: > > > I'm still here :) I haven't coded on the JVM for a couple years so I'm > > just getting started. I bought an IntelliJ license and set up the Spark dev > > environment. Gonna work from both ends - fix some outstanding issues in > > GraphFrames to get more familiar with its internals (somewhat already) and > > dive into GraphX in the Spark project and try to fix a bug or two to start. > > > > Russ > > > > On Tue, Oct 8, 2024 at 3:24 PM Mich Talebzadeh <mich.talebza...@gmail.com> > > wrote: > > > >> GraphFrames is becoming more popular in NLP applications because of its > >> ease of use with PySpark, requiring large-scale data processing. Moreover, > >> Spark DataFrames provide easier integration and manipulation compared to > >> lower-level RDDs in GraphX. However, for performance-critical scenarios, > >> GraphX should still be considered. > >> > >> Mich Talebzadeh, > >> > >> > >> On Tue, 8 Oct 2024 at 23:09, Holden Karau <holden.ka...@gmail.com> wrote: > >> > >>> To be clear the VOTE which passed is simply to deprecate GraphX/warn > >>> people it is not currently being maintained. Before it is removed we would > >>> have another VOTE and plenty of time. If there ends up being a group of > >>> people contributing to GraphX then I think instead of the removal vote > >>> we'll have an un-deprecate vote :) > >>> > >>> On Tue, Oct 8, 2024 at 3:06 PM Russell Jurney <russell.jur...@gmail.com> > >>> wrote: > >>> > >>>> Once GraphFrames is a part of Spark, I would imagine people would come > >>>> forth to create additional MLlib algorithm implementations using > >>>> DataFrames? > >>>> > >>>> I’m gonna start by fixing some bugs and organizing a hackathon. Please > >>>> don’t remove GraphX, I’d like to keep it alive in Spark. I’ll get to work > >>>> this weekend. > >>>> > >>>> On Tue, Oct 8, 2024 at 10:04 AM Mich Talebzadeh < > >>>> mich.talebza...@gmail.com> wrote: > >>>> > >>>>> Short answer > >>>>> > >>>>> - MLlib directly references GraphX because it is part of Spark’s core, > >>>>> but GraphFrames is external and must be added as a dependency manually > >>>>> if > >>>>> you want to use it alongside MLlib. Therefore, you will not see > >>>>> GraphFrames > >>>>> in the pom.xml unless you specifically add it to your project. > >>>>> > >>>>> Long answer > >>>>> > >>>>> GraphX in MLlib: > >>>>> > >>>>> - GraphX is a core part of Apache Spark and is built on top of Spark’s > >>>>> RDD abstraction. It is directly included in the Spark ecosystem, which > >>>>> is > >>>>> why you see GraphX as a dependency in MLlib’s POM file. > >>>>> MLlib uses GraphX primarily for graph-based machine learning > >>>>> algorithms like PageRank, Label Propagation, and Shortest Path. These > >>>>> algorithms are part of the core Spark API and tightly coupled with the > >>>>> RDD-based architecture that GraphX operates on. > >>>>> > >>>>> Why GraphFrames is Not in MLlib’s POM: > >>>>> > >>>>> - GraphFrames is a separate library that builds on top of Spark’s > >>>>> DataFrame API, providing a more flexible and optimized interface for > >>>>> graph > >>>>> processing than GraphX. > >>>>> > >>>>> - GraphFrames is not part of Spark’s core project like GraphX, which > >>>>> is why it's not listed in MLlib's pom.xml. Instead, it’s an external > >>>>> package that you need to install separately (from the Spark Packages > >>>>> repository or Maven Central).If you want to use GraphFrames in a > >>>>> project, > >>>>> you would typically add it as a dependency manually, but it is not > >>>>> bundled > >>>>> directly with MLlib or Spark. > >>>>> > >>>>> Reason for GraphX in MLlib: > >>>>> > >>>>> - MLlib was initially designed around Spark’s core components, which > >>>>> included GraphX as the primary API for graph processing. > >>>>> The direct reference to GraphX in MLlib’s pom.xml ensures > >>>>> compatibility with existing graph-based algorithms that are built into > >>>>> Spark, whereas GraphFrames is a more recent extension and externalized > >>>>> as a > >>>>> package. > >>>>> > >>>>> Why GraphFrames isn't included directly: > >>>>> > >>>>> - GraphFrames is treated as an extension because it represents a more > >>>>> modern approach *that works with DataFrames rather than RDDs*, which > >>>>> is a shift from the earlier RDD-based APIs like GraphX. > >>>>> Since MLlib is part of the core Spark distribution and GraphFrames is > >>>>> an optional library, it is not referenced directly in MLlib's POM. > >>>>> > >>>>> HTH > >>>>> > >>>>> > >>>>> Mich Talebzadeh, > >>>>> > >>>>> > >>>>> *Disclaimer:* The information provided is correct to the best of my > >>>>> knowledge but of course cannot be guaranteed . It is essential to note > >>>>> that, as with any advice, quote "one test result is worth one-thousand > >>>>> expert opinions (Werner > >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun > >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > >>>>> > >>>>> > >>>>> On Tue, 8 Oct 2024 at 16:45, Russell Jurney <russell.jur...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> I was going on the previous comment but I believe only GraphX is in > >>>>>> MLlib. > >>>>>> > >>>>>> On Tue, Oct 8, 2024 at 7:03 AM Ángel <angel.alvarez.pas...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Where is Graphframes referenced in the MLlib's pom? I can't see it, > >>>>>>> however, it does contain the GraphX dependency. > >>>>>>> > >>>>>>> https://github.com/apache/spark/blob/master/mllib/pom.xml > >>>>>>> > >>>>>>> <dependency> > >>>>>>> <groupId>org.apache.spark</groupId> > >>>>>>> <artifactId>*spark-graphx_* > >>>>>>> ${scala.binary.version}</artifactId> > >>>>>>> <version>${project.version}</version> > >>>>>>> </dependency> > >>>>>>> > >>>>>>> El mar, 8 oct 2024 a las 15:45, Russell Jurney (< > >>>>>>> russell.jur...@gmail.com>) escribió: > >>>>>>> > >>>>>>>> GraphFrames is built with the primitives from GraphX. In that case > >>>>>>>> it is: MLlib —> GraphFrames — > GraphX. > >>>>>>>> > >>>>>>>> Here’s GraphFrames.java: > >>>>>>>> > >>>>>>>> https://github.com/graphframes/graphframes/blob/master/src/main/scala/org/graphframes/GraphFrame.scala#L24 > >>>>>>>> > >>>>>>>> I think the most common use of GraphFrames is its connected > >>>>>>>> components library that a lot of people depends on for entity > >>>>>>>> resolution > >>>>>>>> workloads. > >>>>>>>> > >>>>>>>> Russ > >>>>>>>> > >>>>>>>> On Tue, Oct 8, 2024 at 12:22 AM Mich Talebzadeh < > >>>>>>>> mich.talebza...@gmail.com> wrote: > >>>>>>>> > >>>>>>>>> Interesting point > >>>>>>>>> > >>>>>>>>> MLlib primarily uses GraphFrames for graph-based machine learning > >>>>>>>>> tasks, not GraphX? This is because GraphFrames are built on top of > >>>>>>>>> DataFrames, making them more compatible with modern Spark > >>>>>>>>> components like > >>>>>>>>> MLlib, which heavily relies on DataFrames and the Catalyst > >>>>>>>>> optimization > >>>>>>>>> engine for performance improvements. > >>>>>>>>> > >>>>>>>>> HTH > >>>>>>>>> > >>>>>>>>> Mich Talebzadeh, > >>>>>>>>> > >>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime > >>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial > >>>>>>>>> College London > >>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> > >>>>>>>>> London, United Kingdom > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> view my Linkedin profile > >>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> *Disclaimer:* The information provided is correct to the best of > >>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential > >>>>>>>>> to note > >>>>>>>>> that, as with any advice, quote "one test result is worth > >>>>>>>>> one-thousand > >>>>>>>>> expert opinions (Werner > >>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun > >>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Tue, 8 Oct 2024 at 06:22, Ruifeng Zheng <ruife...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 for move GraphFrame into Spark > >>>>>>>>>> > >>>>>>>>>> Besides GraphFrame, MLlib also depends on GraphX. > >>>>>>>>>> > >>>>>>>>>> On Tue, Oct 8, 2024 at 11:20 AM Reynold Xin > >>>>>>>>>> <r...@databricks.com.invalid> wrote: > >>>>>>>>>> > >>>>>>>>>>> We can also consider the following: move GraphFrame into Spark, > >>>>>>>>>>> and make GraphX an internal impl detail of GraphFrame. Then we > >>>>>>>>>>> can over > >>>>>>>>>>> time change the implementation, simplify it (not sure if it is > >>>>>>>>>>> possible, > >>>>>>>>>>> but somebody can look into it).... > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Oct 7, 2024 at 7:04 PM Russell Jurney < > >>>>>>>>>>> russell.jur...@gmail.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Took a look at recent activity. Spark 3.5 support > >>>>>>>>>>>> <https://github.com/graphframes/graphframes/commit/e54f249605dde60787f9b41b88ed7d5872b7dfab> > >>>>>>>>>>>> was > >>>>>>>>>>>> added a year ago. I'm sure we'll add Spark 4 support as soon as > >>>>>>>>>>>> it is out. > >>>>>>>>>>>> > >>>>>>>>>>>> There is a new issue to organize a GraphFrames Hackathon > >>>>>>>>>>>> <https://github.com/graphframes/graphframes/issues/460>. > >>>>>>>>>>>> Please sign up to help! > >>>>>>>>>>>> https://github.com/graphframes/graphframes/issues/460 > >>>>>>>>>>>> > >>>>>>>>>>>> I seriously need GraphX and GraphFrames to make it... I have no > >>>>>>>>>>>> other way of doing property graph motif matching on large > >>>>>>>>>>>> graphs. It's kind > >>>>>>>>>>>> of important to me. > >>>>>>>>>>>> > >>>>>>>>>>>> Some slides on my work with GraphFrames: > >>>>>>>>>>>> > >>>>>>>>>>>> [image: image.png] > >>>>>>>>>>>> > >>>>>>>>>>>> [image: image.png] > >>>>>>>>>>>> > >>>>>>>>>>>> [image: image.png] > >>>>>>>>>>>> > >>>>>>>>>>>> [image: image.png] > >>>>>>>>>>>> > >>>>>>>>>>>> [image: image.png] > >>>>>>>>>>>> > >>>>>>>>>>>> Russell > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Oct 7, 2024 at 6:06 PM Holden Karau < > >>>>>>>>>>>> holden.ka...@gmail.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> That’s awesome! > >>>>>>>>>>>>> > >>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau > >>>>>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ > >>>>>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> > >>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): > >>>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > >>>>>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau > >>>>>>>>>>>>> Pronouns: she/her > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Mon, Oct 7, 2024 at 5:42 PM Russell Jurney < > >>>>>>>>>>>>> russell.jur...@gmail.com> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I’ll organize a hackathon. A friend wants to finish the > >>>>>>>>>>>>>> implementation of Lucian modularity for GraphFrames. I’ll fix > >>>>>>>>>>>>>> some GraphX > >>>>>>>>>>>>>> bugs at it. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I did just blog all about the motif matching in GraphFrames: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> https://blog.graphlet.ai/financial-crime-and-corruption-network-motifs-4cf2e8e10eb5 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Russ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 5:38 PM Holden Karau < > >>>>>>>>>>>>>> holden.ka...@gmail.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> So this discuss thread and the vote thread to deprecate to > >>>>>>>>>>>>>>> leave the option of removing it during 4.X are probably the > >>>>>>>>>>>>>>> highest profile > >>>>>>>>>>>>>>> it’s been in years. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> In the past for parts of Spark I’ve cared about I’ve > >>>>>>>>>>>>>>> organized virtual meetings to co-ordinate work — if your > >>>>>>>>>>>>>>> connected with > >>>>>>>>>>>>>>> some of the Spark+Graph community reaching out to find others > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>> organizing a meeting could be a way to raise the profile a > >>>>>>>>>>>>>>> bit? Maybe > >>>>>>>>>>>>>>> organize a virtual hackathon (I’m meaning to try this for > >>>>>>>>>>>>>>> some other things > >>>>>>>>>>>>>>> so happy to share what I learn from doing that)? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau > >>>>>>>>>>>>>>> Fight Health Insurance: > >>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ > >>>>>>>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> > >>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): > >>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > >>>>>>>>>>>>>>> YouTube Live Streams: > >>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau > >>>>>>>>>>>>>>> Pronouns: she/her > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 5:02 PM Russell Jurney < > >>>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I’ll look for a bug to fix. If GraphX is outside of Spark, > >>>>>>>>>>>>>>>> Spark would tend to break GraphFrames and it will be > >>>>>>>>>>>>>>>> burdensome on an > >>>>>>>>>>>>>>>> external project to keep up. Graph computing on Spark is > >>>>>>>>>>>>>>>> implrtant to a lot > >>>>>>>>>>>>>>>> of people, is there a way to raise visibility here? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 4:24 PM Holden Karau < > >>>>>>>>>>>>>>>> holden.ka...@gmail.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> There are no specific tickets associated with the lack of > >>>>>>>>>>>>>>>>> maintaince or this as the component has not been maintained > >>>>>>>>>>>>>>>>> for a > >>>>>>>>>>>>>>>>> sufficiently long time. If your interested in taking it on > >>>>>>>>>>>>>>>>> that’s > >>>>>>>>>>>>>>>>> wonderful, probably starting with fixing some bugs could be > >>>>>>>>>>>>>>>>> a great place > >>>>>>>>>>>>>>>>> to start and figure out if it’s something you want to do > >>>>>>>>>>>>>>>>> long term. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I would recommend making a first bug fix in a actively > >>>>>>>>>>>>>>>>> maintained area of Spark to get to > >>>>>>>>>>>>>>>>> Know some reviewers since there is not anyone tracking the > >>>>>>>>>>>>>>>>> GraphX PRs. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> As a note I don’t think GraphX is required for Graph > >>>>>>>>>>>>>>>>> Frames long term, so another option would be to talk to the > >>>>>>>>>>>>>>>>> GraphFrames > >>>>>>>>>>>>>>>>> folks and move the GraphX code over to it. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Ideally we’d have someone willing to act as a mentor or > >>>>>>>>>>>>>>>>> guide but so far we have no volunteers (especially no one > >>>>>>>>>>>>>>>>> familiar with the > >>>>>>>>>>>>>>>>> graph X code). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau > >>>>>>>>>>>>>>>>> Fight Health Insurance: > >>>>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ > >>>>>>>>>>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> > >>>>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): > >>>>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > >>>>>>>>>>>>>>>>> YouTube Live Streams: > >>>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau > >>>>>>>>>>>>>>>>> Pronouns: she/her > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 3:25 PM Russell Jurney < > >>>>>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I volunteer to maintain GraphX to keep GraphFrames a > >>>>>>>>>>>>>>>>>> viable project. I don’t have a clear view on whether it > >>>>>>>>>>>>>>>>>> works with Spark 4 > >>>>>>>>>>>>>>>>>> or if it needs updates? I don’t have Spark commits but I’m > >>>>>>>>>>>>>>>>>> a committer on > >>>>>>>>>>>>>>>>>> Apache DataFu and mentored the Spark feature for it. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Can someone tell me what is involved? Point me at a > >>>>>>>>>>>>>>>>>> ticket? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Russell > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Mon, Oct 7, 2024 at 12:11 AM Erik Eklund < > >>>>>>>>>>>>>>>>>> eekl...@definitivehc.com> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hello, > >>>>>>>>>>>>>>>>>>> We rely on GraphX for an important component of our > >>>>>>>>>>>>>>>>>>> product. And we really want it to stay a typed interface. > >>>>>>>>>>>>>>>>>>> Please keep > >>>>>>>>>>>>>>>>>>> GraphX. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Erik > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> *From: *Holden Karau <holden.ka...@gmail.com> > >>>>>>>>>>>>>>>>>>> *Date: *Sunday, October 6, 2024 at 06:22 > >>>>>>>>>>>>>>>>>>> *To: *Ángel <angel.alvarez.pas...@gmail.com> > >>>>>>>>>>>>>>>>>>> *Cc: *Russell Jurney <russell.jur...@gmail.com>, Mich > >>>>>>>>>>>>>>>>>>> Talebzadeh <mich.talebza...@gmail.com>, Spark dev list < > >>>>>>>>>>>>>>>>>>> dev@spark.apache.org>, user @spark < > >>>>>>>>>>>>>>>>>>> u...@spark.apache.org> > >>>>>>>>>>>>>>>>>>> *Subject: *Re: [DISCUSS] Deprecate GraphX OR Find new > >>>>>>>>>>>>>>>>>>> maintainers interested in GraphX OR leave it as is? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> So are there companies using it? And are they willing to > >>>>>>>>>>>>>>>>>>> contribute to maintaining it? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Fight Health Insurance: > >>>>>>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ > >>>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f%3fq%3dhk_email&c=E,1,OT9ylxCx5xRNCToPSzu0VEvefs4uts16fTBydH2NiLHMGEwLjrEXgkhU8W-Ai6xD8VDMyWea44GBMOEecMNdapaZKZbBTrZpquOBKi6YRlqu-FVAzji6-w,,&typo=1> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): > >>>>>>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 > >>>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,h0ccgHctUPRY4zAN_qZ-qdBgLDpQLtm7KaOL4u12U4PR7PeJ4MUBOS8bbD7CNssUIMqRMvY_pOqbh7PfLY0lRpQh9mfqBC0KnSHBZzxxSJJr-55r5kv6YjYwrA,,&typo=1> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> YouTube Live Streams: > >>>>>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Pronouns: she/her > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 9:17 PM Ángel < > >>>>>>>>>>>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> That would definitely affect companies using GraphX, but > >>>>>>>>>>>>>>>>>>> at least they’d have the choice to migrate their code. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I think that’s probably the way to go. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> El dom, 6 oct 2024 a las 6:09, Holden Karau (< > >>>>>>>>>>>>>>>>>>> holden.ka...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> So removing GraphX from Spark would not prevent > >>>>>>>>>>>>>>>>>>> GraphFrames from continuing, they could pick up the > >>>>>>>>>>>>>>>>>>> GraphX source and > >>>>>>>>>>>>>>>>>>> incorporate it into their project. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Fight Health Insurance: > >>>>>>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ > >>>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f%3fq%3dhk_email&c=E,1,9xMMQlY7gtmkqxT0NTmS8KMg4wOUjw0PWKM-oepAYAkE-SiM5pyXCb80AuRZYJ4zMIedVlwVMAKi_eh52Hof0LsteXx2eIslnsDBdmVeuocpILpneg,,&typo=1> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): > >>>>>>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 > >>>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,kbGbMBRMidAYi0aqUmj949vRahpEjVzSgJv_YYtO5EteSXZy4RrMYXJU48mN2CyS5sdovsgiFAAiBLnyQ29gCCn8xbTrEJmfIhjtH7tD4N31VUoLtQ,,&typo=1> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> YouTube Live Streams: > >>>>>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Pronouns: she/her > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 5:22 PM Russell Jurney < > >>>>>>>>>>>>>>>>>>> russell.jur...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> A lot of people like me use GraphFrames for its > >>>>>>>>>>>>>>>>>>> connected components implementation and its motif > >>>>>>>>>>>>>>>>>>> matching feature. I am > >>>>>>>>>>>>>>>>>>> willing to work on it to keep it alive. They did a 0.8.3 > >>>>>>>>>>>>>>>>>>> release not too > >>>>>>>>>>>>>>>>>>> long ago. Please keep GraphX alive. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Sat, Oct 5, 2024 at 3:44 PM Mich Talebzadeh < > >>>>>>>>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I added the user list as they may have vested > >>>>>>>>>>>>>>>>>>> interest here and and hopefully can contribute > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Few suggestions: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 1. Data-Driven Decision Making: Return to the core > >>>>>>>>>>>>>>>>>>> metrics—analyze usage trends, performance benchmarks, > >>>>>>>>>>>>>>>>>>> and the actual impact > >>>>>>>>>>>>>>>>>>> on businesses that rely on GraphX. Objectivity can be > >>>>>>>>>>>>>>>>>>> restored by letting > >>>>>>>>>>>>>>>>>>> data speak louder than opinions so to speak. > >>>>>>>>>>>>>>>>>>> 2. Broaden the Discussion: Engage more stakeholders > >>>>>>>>>>>>>>>>>>> from diverse backgrounds (especially spark users) to > >>>>>>>>>>>>>>>>>>> bring in new > >>>>>>>>>>>>>>>>>>> perspectives and counterbalance the more vocal but > >>>>>>>>>>>>>>>>>>> potentially narrow > >>>>>>>>>>>>>>>>>>> interests of core maintainers or open-source > >>>>>>>>>>>>>>>>>>> contributors. > >>>>>>>>>>>>>>>>>>> 3. Define Clear Criteria for Decision Making: Agree > >>>>>>>>>>>>>>>>>>> on a set of objective criteria by which the project’s > >>>>>>>>>>>>>>>>>>> future will be > >>>>>>>>>>>>>>>>>>> judged. These could include market demand, > >>>>>>>>>>>>>>>>>>> contribution levels, maintenance > >>>>>>>>>>>>>>>>>>> costs, alternative solutions, and alignment with the > >>>>>>>>>>>>>>>>>>> overall Spark > >>>>>>>>>>>>>>>>>>> ecosystem goals. Some have already been covered. > >>>>>>>>>>>>>>>>>>> 4. Timely Conclusion of Discussions: Set a timeline > >>>>>>>>>>>>>>>>>>> for making a decision. Long, open-ended discussions > >>>>>>>>>>>>>>>>>>> tend to lose focus. > >>>>>>>>>>>>>>>>>>> Putting deadlines forces participants to focus on key > >>>>>>>>>>>>>>>>>>> issues and prevents > >>>>>>>>>>>>>>>>>>> endless debates. > >>>>>>>>>>>>>>>>>>> 5. Borrowing from commercial settings, it is often > >>>>>>>>>>>>>>>>>>> necessary for a strong leadership team to step in and > >>>>>>>>>>>>>>>>>>> make the final > >>>>>>>>>>>>>>>>>>> decision after considering the input. When the > >>>>>>>>>>>>>>>>>>> objectivity of discussions > >>>>>>>>>>>>>>>>>>> starts to wane, leadership needs to cut through the > >>>>>>>>>>>>>>>>>>> round discussions and > >>>>>>>>>>>>>>>>>>> steer towards action based on business and technical > >>>>>>>>>>>>>>>>>>> realities. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> HTH > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Mich Talebzadeh, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Architect | Data Engineer | Data Science | Financial > >>>>>>>>>>>>>>>>>>> Crime > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> > >>>>>>>>>>>>>>>>>>> Imperial College London > >>>>>>>>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> London, United Kingdom > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> [image: Image removed by sender.] view my Linkedin > >>>>>>>>>>>>>>>>>>> profile > >>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh > >>>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fen.everybodywiki.com%2fMich_Talebzadeh&c=E,1,U1JaGVMkko53HkJO5fwmkIXfziTOWL3K1CkAeHwFG55TbZQUd5xVNLGpLt2o0ytujE6zaLpqU2GWCZqHSbo3SU4Wh9Rl8NG4bWPbFWUwyw,,&typo=1> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to > >>>>>>>>>>>>>>>>>>> the best of my knowledge but of course cannot be > >>>>>>>>>>>>>>>>>>> guaranteed . It is > >>>>>>>>>>>>>>>>>>> essential to note that, as with any advice, quote "one > >>>>>>>>>>>>>>>>>>> test > >>>>>>>>>>>>>>>>>>> result is worth one-thousand expert opinions (Werner > >>>>>>>>>>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > >>>>>>>>>>>>>>>>>>> Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun> > >>>>>>>>>>>>>>>>>>> )". > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Sat, 5 Oct 2024 at 06:26, Ángel < > >>>>>>>>>>>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I completely agree with everyone here. I don’t think the > >>>>>>>>>>>>>>>>>>> issue is deprecating it; to me, the problem lies in not > >>>>>>>>>>>>>>>>>>> providing a new and > >>>>>>>>>>>>>>>>>>> better solution for handling graphs in Spark. In the > >>>>>>>>>>>>>>>>>>> past, I used GraphX > >>>>>>>>>>>>>>>>>>> via GraphFrames for record linkage, and I found it both > >>>>>>>>>>>>>>>>>>> useful and > >>>>>>>>>>>>>>>>>>> effective. Is there any discussion about a potential > >>>>>>>>>>>>>>>>>>> replacement? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I’d be willing to help maintain GraphX, though I don’t > >>>>>>>>>>>>>>>>>>> have previous experience with maintaining open-source > >>>>>>>>>>>>>>>>>>> projects. All I can > >>>>>>>>>>>>>>>>>>> promise is good intentions, willingness to learn and lots > >>>>>>>>>>>>>>>>>>> of energy and > >>>>>>>>>>>>>>>>>>> passion. Is that enough? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Btw, what's your take on this? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> · *GraphX* will be deprecated in favor of a new > >>>>>>>>>>>>>>>>>>> graphing component, SparkGraph, based on Cypher > >>>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fneo4j.com%2fdeveloper%2fcypher-query-language%2f&c=E,1,5sP_K0oxQDLYIfWhFPwgNEmTuXMR7tvCjLLcf_ZBAv7oIBySxARy9TyrqNkmZKfXwrIDrhe6TVBCUun2luRV_mAbSD4rooD9YRt5GYYgbHbBUYerg1mpA4Oe6eo,&typo=1>, > >>>>>>>>>>>>>>>>>>> a much richer graph language than previously offered by > >>>>>>>>>>>>>>>>>>> GraphX. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> https://cloud.google.com/blog/products/data-analytics/introducing-spark-3-and-hadoop-3-on-dataproc-image-version-2-0 > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> El sáb, 5 oct 2024 a las 2:17, Mark Hamstra (< > >>>>>>>>>>>>>>>>>>> markhams...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> As I wrote to Holden privately, I might well change my > >>>>>>>>>>>>>>>>>>> vote to be in > >>>>>>>>>>>>>>>>>>> favor of a deprecation label combined with some > >>>>>>>>>>>>>>>>>>> effective means of > >>>>>>>>>>>>>>>>>>> communicating that this doesn't mean the end for GraphX > >>>>>>>>>>>>>>>>>>> if interested > >>>>>>>>>>>>>>>>>>> contributors come forward to rescue it. I don't like > >>>>>>>>>>>>>>>>>>> either the idea > >>>>>>>>>>>>>>>>>>> of keeping unmaintained code and public APIs around > >>>>>>>>>>>>>>>>>>> (especially if > >>>>>>>>>>>>>>>>>>> there are problems with them) or the idea of removing > >>>>>>>>>>>>>>>>>>> Spark > >>>>>>>>>>>>>>>>>>> functionality just because no one has contributed to it > >>>>>>>>>>>>>>>>>>> for a while. A > >>>>>>>>>>>>>>>>>>> naked deprecation label feels somewhat drastic and > >>>>>>>>>>>>>>>>>>> pre-emptive to me. > >>>>>>>>>>>>>>>>>>> I don't expect that GraphX will be the last part of > >>>>>>>>>>>>>>>>>>> Spark to run the > >>>>>>>>>>>>>>>>>>> risk of death through neglect, and I think we need an > >>>>>>>>>>>>>>>>>>> effective means > >>>>>>>>>>>>>>>>>>> of encouraging resuscitation that a deprecation label on > >>>>>>>>>>>>>>>>>>> its own does > >>>>>>>>>>>>>>>>>>> not provide. On the other hand, if no one really is > >>>>>>>>>>>>>>>>>>> willing to come to > >>>>>>>>>>>>>>>>>>> the aid of GraphX or other neglected functionality given > >>>>>>>>>>>>>>>>>>> adequate > >>>>>>>>>>>>>>>>>>> warning of possible removal, I'm not then opposed to the > >>>>>>>>>>>>>>>>>>> usual > >>>>>>>>>>>>>>>>>>> deprecation and removal process. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Fri, Oct 4, 2024 at 4:10 PM Sean Owen < > >>>>>>>>>>>>>>>>>>> sro...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > This is a reasonable discussion, but maybe the more > >>>>>>>>>>>>>>>>>>> practical point is: are you sure you want to block this > >>>>>>>>>>>>>>>>>>> unilaterally? This > >>>>>>>>>>>>>>>>>>> effectively makes a decision that GraphX cannot be > >>>>>>>>>>>>>>>>>>> removed for a long > >>>>>>>>>>>>>>>>>>> while. I'd understand it more if we had an active > >>>>>>>>>>>>>>>>>>> maintainer and/or active > >>>>>>>>>>>>>>>>>>> user proposing to veto, but my understanding is this is > >>>>>>>>>>>>>>>>>>> just a proposal to > >>>>>>>>>>>>>>>>>>> block this on behalf of some users, someone else who > >>>>>>>>>>>>>>>>>>> might do some work and > >>>>>>>>>>>>>>>>>>> hasn't to date for some reason. Add to that the fact that > >>>>>>>>>>>>>>>>>>> the 'pro' > >>>>>>>>>>>>>>>>>>> arguments all seem to be arguments for working on > >>>>>>>>>>>>>>>>>>> GraphFrames, and I find > >>>>>>>>>>>>>>>>>>> this somewhat drastic. > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > On Fri, Oct 4, 2024 at 5:23 PM Mark Hamstra < > >>>>>>>>>>>>>>>>>>> markhams...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>> >> "You can't say nothing is removable until there are > >>>>>>>>>>>>>>>>>>> no users." > >>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>> >> That is not what I am saying. Rather, I am countering > >>>>>>>>>>>>>>>>>>> what others seem > >>>>>>>>>>>>>>>>>>> >> to be suggesting: There are no users and no interest, > >>>>>>>>>>>>>>>>>>> therefore we can > >>>>>>>>>>>>>>>>>>> >> and should deprecate. > >>>>>>>>>>>>>>>>>>> >> > >>>>>>>>>>>>>>>>>>> >> On Fri, Oct 4, 2024 at 3:10 PM Sean Owen < > >>>>>>>>>>>>>>>>>>> sro...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>>> >> > I could flip this argument around. More strongly, > >>>>>>>>>>>>>>>>>>> not being deprecated means "won't be removed" and > >>>>>>>>>>>>>>>>>>> likewise implies support > >>>>>>>>>>>>>>>>>>> and development. I don't think either of the latter have > >>>>>>>>>>>>>>>>>>> been true for > >>>>>>>>>>>>>>>>>>> years. What suggests this will change? A todo list is not > >>>>>>>>>>>>>>>>>>> going to do > >>>>>>>>>>>>>>>>>>> anything, IMHO. > >>>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>>> >> > I'm also concerned about the cost of that, which I > >>>>>>>>>>>>>>>>>>> have observed. GraphX PRs are almost certainly not going > >>>>>>>>>>>>>>>>>>> to be reviewed > >>>>>>>>>>>>>>>>>>> because of its state. Deprecation both communicates that > >>>>>>>>>>>>>>>>>>> reality, and > >>>>>>>>>>>>>>>>>>> leaves an option open, whereas not deprecating forecloses > >>>>>>>>>>>>>>>>>>> that option for a > >>>>>>>>>>>>>>>>>>> while. > >>>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>>> >> > I don't think the question is, does anyone use it? > >>>>>>>>>>>>>>>>>>> because anyone can continue to use it -- in Spark 3.x for > >>>>>>>>>>>>>>>>>>> sure, and in 4.x > >>>>>>>>>>>>>>>>>>> if not removed. > >>>>>>>>>>>>>>>>>>> >> > You can't say nothing is removable until there are > >>>>>>>>>>>>>>>>>>> no users. > >>>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>>> >> > Also, why would GraphFrames not be the logical home > >>>>>>>>>>>>>>>>>>> of this going forward anyway? which I think is the > >>>>>>>>>>>>>>>>>>> subtext. > >>>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>>> >> > On Fri, Oct 4, 2024 at 4:56 PM Mark Hamstra < > >>>>>>>>>>>>>>>>>>> markhams...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>>> >> >> I'm -1(*) because, while it technically means > >>>>>>>>>>>>>>>>>>> "might be removed in the > >>>>>>>>>>>>>>>>>>> >> >> future", I think developers and users are more > >>>>>>>>>>>>>>>>>>> prone to interpret > >>>>>>>>>>>>>>>>>>> >> >> something being marked as deprecated as "very > >>>>>>>>>>>>>>>>>>> likely will be removed > >>>>>>>>>>>>>>>>>>> >> >> in the future, so don't depend on this or waste > >>>>>>>>>>>>>>>>>>> your time contributing > >>>>>>>>>>>>>>>>>>> >> >> to its further development." I don't think the > >>>>>>>>>>>>>>>>>>> latter is what we want > >>>>>>>>>>>>>>>>>>> >> >> just because something hasn't been updated > >>>>>>>>>>>>>>>>>>> meaningfully in a while. > >>>>>>>>>>>>>>>>>>> >> >> There have been How To articles for GraphX and > >>>>>>>>>>>>>>>>>>> Graph Frames posted in > >>>>>>>>>>>>>>>>>>> >> >> the not too distant past, and the Google Search > >>>>>>>>>>>>>>>>>>> trend shows a pretty > >>>>>>>>>>>>>>>>>>> >> >> steady level of interest, not a decline to zero, > >>>>>>>>>>>>>>>>>>> so I don't think that > >>>>>>>>>>>>>>>>>>> >> >> it is accurate to declare that there is no use or > >>>>>>>>>>>>>>>>>>> interest in GraphX. > >>>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>>> >> >> Unless retaining GraphX is imposing significant > >>>>>>>>>>>>>>>>>>> costs on continuing > >>>>>>>>>>>>>>>>>>> >> >> Spark development, I can't support deprecating > >>>>>>>>>>>>>>>>>>> GraphX. I can support > >>>>>>>>>>>>>>>>>>> >> >> encouraging GraphX and Graph Frames development > >>>>>>>>>>>>>>>>>>> through something like > >>>>>>>>>>>>>>>>>>> >> >> a To Do list or document of "What we'd like to see > >>>>>>>>>>>>>>>>>>> in the way of > >>>>>>>>>>>>>>>>>>> >> >> further development of Spark's graph processing > >>>>>>>>>>>>>>>>>>> capabilities" -- i.e., > >>>>>>>>>>>>>>>>>>> >> >> things that encourage and support new > >>>>>>>>>>>>>>>>>>> contributions to address any > >>>>>>>>>>>>>>>>>>> >> >> shortcomings in Spark's graph processing, not > >>>>>>>>>>>>>>>>>>> things that discourage > >>>>>>>>>>>>>>>>>>> >> >> contributions and use in the way that I believe > >>>>>>>>>>>>>>>>>>> simply declaring > >>>>>>>>>>>>>>>>>>> >> >> GraphX to be deprecated would. > >>>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>>> >> >> On Sun, Sep 29, 2024 at 11:04 AM Holden Karau < > >>>>>>>>>>>>>>>>>>> holden.ka...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>>> >> >> > > >>>>>>>>>>>>>>>>>>> >> >> > Since we're getting close to cutting a 4.0 > >>>>>>>>>>>>>>>>>>> branch I'd like to float the idea of officially > >>>>>>>>>>>>>>>>>>> deprecating Graph X. What > >>>>>>>>>>>>>>>>>>> that would mean (to me) is we would update the docs to > >>>>>>>>>>>>>>>>>>> indicate that Graph > >>>>>>>>>>>>>>>>>>> X is deprecated and it's APIs may be removed at anytime > >>>>>>>>>>>>>>>>>>> in the future. > >>>>>>>>>>>>>>>>>>> >> >> > > >>>>>>>>>>>>>>>>>>> >> >> > Alternatively, we could mark it as "unmaintained > >>>>>>>>>>>>>>>>>>> and in search of maintainers" with a note that if no > >>>>>>>>>>>>>>>>>>> maintainers are found, > >>>>>>>>>>>>>>>>>>> we may remove it in a future minor version. > >>>>>>>>>>>>>>>>>>> >> >> > > >>>>>>>>>>>>>>>>>>> >> >> > Looking at the source graph X, I don't see any > >>>>>>>>>>>>>>>>>>> meaningful active development going back over three > >>>>>>>>>>>>>>>>>>> years*. There is even a > >>>>>>>>>>>>>>>>>>> thread on user@ from 2017 asking if graph X is > >>>>>>>>>>>>>>>>>>> maintained anymore, with no response from the developers. > >>>>>>>>>>>>>>>>>>> >> >> > > >>>>>>>>>>>>>>>>>>> >> >> > Now I'm open to the idea that GraphX is stable > >>>>>>>>>>>>>>>>>>> and "works as is" and simply doesn't require > >>>>>>>>>>>>>>>>>>> modifications but given the > >>>>>>>>>>>>>>>>>>> user thread I'm a little concerned here about bringing > >>>>>>>>>>>>>>>>>>> this API with us > >>>>>>>>>>>>>>>>>>> into Spark 4 if we don't have anyone signed up to > >>>>>>>>>>>>>>>>>>> maintain it. > >>>>>>>>>>>>>>>>>>> >> >> > > >>>>>>>>>>>>>>>>>>> >> >> > * Excluding globally applied changes > >>>>>>>>>>>>>>>>>>> >> >> > -- > >>>>>>>>>>>>>>>>>>> >> >> > Twitter: https://twitter.com/holdenkarau > >>>>>>>>>>>>>>>>>>> >> >> > Fight Health Insurance: > >>>>>>>>>>>>>>>>>>> https://www.fighthealthinsurance.com/ > >>>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f&c=E,1,9CeJ-bKUShnxOFZMc15zJG1qgfAB9rnSDzrmLzNiXb8qE0NXedNCoZy4HobcS7laOMqtvJzYjvDzjBld1FaCPZpOBW6cf1l_xaG4bEbjYoDpNG0zuQ9_K5TW&typo=1> > >>>>>>>>>>>>>>>>>>> >> >> > Books (Learning Spark, High Performance Spark, > >>>>>>>>>>>>>>>>>>> etc.): https://amzn.to/2MaRAG9 > >>>>>>>>>>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,HJPBNbN3nfUZcb0-2OgveqIE5I5lvPSv-bOfRXIprFdSsGMlNq15o6rueLf2ZQRfytMu0-t3IxSjYou2uuPzUrSAqJ0LV42n2hG8rnkkpN4AA5w4mQZFTs4,&typo=1> > >>>>>>>>>>>>>>>>>>> >> >> > YouTube Live Streams: > >>>>>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau > >>>>>>>>>>>>>>>>>>> >> >> > Pronouns: she/her > >>>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- > >>>>>>>>>>>>>>>>>>> >> >> To unsubscribe e-mail: > >>>>>>>>>>>>>>>>>>> dev-unsubscr...@spark.apache.org > >>>>>>>>>>>>>>>>>>> >> >> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- > >>>>>>>>>>>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>> > >>> -- > >>> Twitter: https://twitter.com/holdenkarau > >>> Fight Health Insurance: https://www.fighthealthinsurance.com/ > >>> <https://www.fighthealthinsurance.com/?q=hk_email> > >>> Books (Learning Spark, High Performance Spark, etc.): > >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau > >>> Pronouns: she/her > >>> > >> > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
