Can you please share the code? It doesn't seem an ideal solution, but if AQE is confused, disabling it makes sense. I can't figure out why a low partition count for an 8 node, 6 edge network would require a lot of partitions... users may have different numbers... do you suggest we enforce some minimum partition count?
On Tue, Jan 14, 2025 at 7:33 AM Ángel <angel.alvarez.pas...@gmail.com> wrote: > Are you sure that temporarily disabling a global setting like AQE is the > best approach to fix this issue? > I increased the number of shuffle partitions in the Spark session > configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the > "checkpoint interval" test ran perfectly without throwing an OOM error. > Why? No idea, but it worked. > > > > El lun, 13 ene 2025 a las 16:45, Russell Jurney (<russell.jur...@gmail.com>) > escribió: > >> Merged, thanks guys! >> >> Russ >> >> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <bjornjorgen...@gmail.com> >> wrote: >> >>> I tested stuff in this PR >>> https://github.com/graphframes/graphframes/pull/477 and then I made >>> this PR https://github.com/graphframes/graphframes/pull/478 >>> >>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <angel.alvarez.pas...@gmail.com >>> >: >>> >>>> Hi Russell, >>>> >>>> I've just got the OOM error during Test 13. I'm running it from >>>> IntelliJ on Windows with Java 11. >>>> >>>> [image: image.png] >>>> I'll look into it over the course of the next week. >>>> >>>> Regards, >>>> Ángel >>>> >>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (< >>>> russell.jur...@gmail.com>) escribió: >>>> >>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I have a >>>>> question for you... >>>>> >>>>> I can't get the unit test 'two components and two dangling vertices' >>>>> in the org.graphframes.lib.ConnectedComponentsSuite >>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148> >>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am >>>>> a >>>>> little stuck on completing a docs release with a motif finding >>>>> tutorial <https://github.com/graphframes/graphframes/pull/473> due to >>>>> this issue. >>>>> >>>>> The problem is outlined in this gist: >>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf >>>>> >>>>> Can someone else please try this and see if it passes on the master >>>>> branch? >>>>> >>>>> > build/sbt clean compile package test >>>>> >>>>> I've tried giving it lots of RAM just to see if it would help, as much >>>>> as 32g driver and 16g for executors and... it has no effect. The test >>>>> graph >>>>> is 8 nodes and 6 edges >>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>, >>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of >>>>> my CPU get used, it spikes as indicated in the image in the gist. >>>>> >>>>> I am running the following setup: >>>>> >>>>> * Ubuntu 20.04 (22.04 in the Docker image) >>>>> * OpenJDK 11 (I also tried 8, same problem) >>>>> * Scala 2.12.20 (I also tried 2.13, same problem) >>>>> * Python 3.11 (I also tried 3.9, same problem) >>>>> >>>>> Or I am running the Dockerfile in the gist >>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile> >>>>> . >>>>> >>>>> Any help much appreciated! Thanks >>>>> >>>>> ----------------------------------------------------------------- >>>>> Oh, some new community stuff for GraphFrames. Hackathon announced next >>>>> week :) >>>>> >>>>> >>>>> - GraphFrames Mailing List >>>>> <https://groups.google.com/g/graphframes/>: ask questions about >>>>> GraphFrames on our Google Group >>>>> - #graphframes Discord Channel on GraphGeeks >>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674> >>>>> >>>>> Thanks! >>>>> Russell Jurney @rjurney <http://twitter.com/rjurney> >>>>> russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB >>>>> <http://facebook.com/jurney> datasyndrome.com >>>>> >>>> >>> >>> -- >>> Bjørn Jørgensen >>> Vestre Aspehaug 4, 6010 Ålesund >>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g> >>> Norge >>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g> >>> >>> +47 480 94 297 >>> >>
