dcausse claimed this task.
dcausse moved this task from Ready for Development to Needs Reporting on the
Discovery-Search (Current work) board.
dcausse added a comment.
I ran a backfill (reaching directly to appservers) using a thread pool of
size 6 over 12 workers (72) and the impact on the app servers was barely
noticeable. We can somehow control the parallelism using the options of the
flink pipeline itself. The pipeline has been running from yarn with these
values for a couple months now so I'm tentatively calling this done.
We can reconsider using other techniques like our poolcounter but I doubt
this is worth the effort at this point.
Related pipeline options:
parallelism: 12
wikibase_repo_thread_pool_size: 6
TASK DETAIL
https://phabricator.wikimedia.org/T275133
WORKBOARD
https://phabricator.wikimedia.org/project/board/1227/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: dcausse
Cc: dcausse, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, wkandek,
JMeybohm, CBogen, Akuckartz, Nandana, Namenlos314, jijiki, Lahi, Gq86,
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll,
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, Dzahn
_______________________________________________
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
