AndrewTavis_WMDE added a comment.
@dcausse, a general point on my end is that when I'm trying to run the code
that you sent along via an HTML on `people.wikimedia.org` I'm getting the
following as an output of Spark runs repeated over and over again:
23/07/31 13:01:58 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Requesting driver to remove executor 1 for reason Container killed by YARN for
exceeding physical memory limits. 4.4 GB of 4.4 GB physical memory used.
Consider boosting spark.executor.memoryOverhead.
This seems to be happening given your `create_custom_session` setup, and
doesn't happen when I do normal `create_session` as seen below:
spark_session = wmf.spark.create_session(type='yarn-large',
app_name="wdqs-subgraph-analysis")
Would you be able to let me know if there's something in my permissions or
setup that's causing this? I'm assuming that your setup will make queries
faster, but we can disregard if my working setup gets me mostly there. I'm
running Jupyter on `stat1005`, and saw that AKhatun was using `stat1008`, in
case that's helpful information :)
TASK DETAIL
https://phabricator.wikimedia.org/T342111
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Gehel, dr0ptp4kt, AndrewTavis_WMDE, Aklapper,
Manuel, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot,
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic,
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude,
Mbch331
_______________________________________________
Wikidata-bugs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
