EBernhardson added a comment.
Stats on the final join building `topSubgraphTriples`. this is using 4096 partitions and repartition(). It works for now so probably not worth dealing with the skew, but these stats might be useful to compare against in the future if it starts failing: | Metric | Min | 25th percentile | Median | 75th percentile | Max | | Duration | 15 s | 46 s | 54 s | 1.0 min | 9.2 min | | Scheduler Delay | 2 ms | 3 ms | 3 ms | 4 ms | 0.4 s | | Task Deserialization Time | 1 ms | 2 ms | 2 ms | 3 ms | 0.7 s | | GC Time | 27 ms | 0.1 s | 0.2 s | 0.3 s | 41 s | | Result Serialization Time | 0 ms | 0 ms | 0 ms | 0 ms | 1 ms | | Getting Result Time | 0 ms | 0 ms | 0 ms | 0 ms | 0 ms | | Peak Execution Memory | 2.1 GB | 2.1 GB | 2.1 GB | 2.1 GB | 13.6 GB | | Shuffle Read Blocked Time | 0 ms | 23 s | 32 s | 38 s | 2.1 min | | Shuffle Read Size / Records | 263.2 MB / 3156075 | 269.9 MB / 3235843 | 271.6 MB / 3256300 | 273.4 MB / 3277774 | 30.5 GB / 414401248 | | Shuffle Remote Reads | 255.2 MB | 264.1 MB | 266.1 MB | 268.0 MB | 29.7 GB | | Shuffle Write Size / Records | 340.9 MB / 3184514 | 351.8 MB / 3281889 | 354.4 MB / 3305742 | 357.0 MB / 3330833 | 367.5 MB / 3438583 | | Shuffle spill (memory) | 0.0 B | 0.0 B | 0.0 B | 0.0 B | 98.1 GB | | Shuffle spill (disk) | 0.0 B | 0.0 B | 0.0 B | 0.0 B | 28.2 GB | | TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AKhatun_WMF, EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Hellket777, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
