I tested stuff in this PR https://github.com/graphframes/graphframes/pull/477 and then I made this PR https://github.com/graphframes/graphframes/pull/478
søn. 12. jan. 2025 kl. 23:10 skrev Ángel <angel.alvarez.pas...@gmail.com>: > Hi Russell, > > I've just got the OOM error during Test 13. I'm running it from IntelliJ > on Windows with Java 11. > > [image: image.png] > I'll look into it over the course of the next week. > > Regards, > Ángel > > El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<russell.jur...@gmail.com>) > escribió: > >> Friends of GraphFrames (github.com/graphframes/graphframes), I have a >> question for you... >> >> I can't get the unit test 'two components and two dangling vertices' in >> the org.graphframes.lib.ConnectedComponentsSuite >> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148> >> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a >> little stuck on completing a docs release with a motif finding tutorial >> <https://github.com/graphframes/graphframes/pull/473> due to this issue. >> >> The problem is outlined in this gist: >> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf >> >> Can someone else please try this and see if it passes on the master >> branch? >> >> > build/sbt clean compile package test >> >> I've tried giving it lots of RAM just to see if it would help, as much as >> 32g driver and 16g for executors and... it has no effect. The test graph is 8 >> nodes and 6 edges >> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>, >> so it shouldn't have a memory problem... yet when it runs, all 24 cores of >> my CPU get used, it spikes as indicated in the image in the gist. >> >> I am running the following setup: >> >> * Ubuntu 20.04 (22.04 in the Docker image) >> * OpenJDK 11 (I also tried 8, same problem) >> * Scala 2.12.20 (I also tried 2.13, same problem) >> * Python 3.11 (I also tried 3.9, same problem) >> >> Or I am running the Dockerfile in the gist >> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile> >> . >> >> Any help much appreciated! Thanks >> >> ----------------------------------------------------------------- >> Oh, some new community stuff for GraphFrames. Hackathon announced next >> week :) >> >> >> - GraphFrames Mailing List <https://groups.google.com/g/graphframes/>: >> ask questions about GraphFrames on our Google Group >> - #graphframes Discord Channel on GraphGeeks >> <https://discord.com/channels/1162999022819225631/1326257052368113674> >> >> Thanks! >> Russell Jurney @rjurney <http://twitter.com/rjurney> >> russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB >> <http://facebook.com/jurney> datasyndrome.com >> > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
