[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-18 Thread mobrovac 
mobrovac added a comment.

As @ori mentioned, the https://phabricator.wikimedia.org/tag/services/ team is 
working with https://phabricator.wikimedia.org/tag/analytics/ on the 
https://phabricator.wikimedia.org/tag/eventbus/ project, whose aim is basically 
to replace jobrunners and enable efficient event propagation and delivery 
inside the production cluster (but also targeted at third-party users). In that 
context, our team plans to build change-propagation system (cf. 
https://phabricator.wikimedia.org/T102476, 
https://phabricator.wikimedia.org/T117933 and related tasks for more context) 
on top of it. And from the looks of it, that's exactly what you are trying to 
achieve with this script, isn't it?


TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac
Cc: Tobi_WMDE_SW, ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, 
aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, 
Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-18 Thread ReleaseTaggerBot 
ReleaseTaggerBot added a project: WMF-deploy-2015-11-10_(1.27.0-wmf.6).

TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ReleaseTaggerBot
Cc: Tobi_WMDE_SW, ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, 
aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, 
Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-18 Thread gerritbot 
gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Tobi_WMDE_SW, ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, 
aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, 
Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-18 Thread ori 
ori removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ori
Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, 
Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-18 Thread ori 
ori added a blocking task: T48643: [Story] Dispatching via delayed jobs 
(instead of cron script).

TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ori
Cc: ori, mobrovac, thiemowmde, aaron, jcrespo, gerritbot, daniel, aude, hoo, 
Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-10 Thread ReleaseTaggerBot 
ReleaseTaggerBot added projects: MW-1.27-release-notes, 
WMF-deploy-2015-11-17_(1.27.0-wmf.7).

TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ReleaseTaggerBot
Cc: aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, 
Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-10 Thread gerritbot 
gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, 
Wikidata-bugs, Mbch331, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-10 Thread Lydia_Pintscher 
Lydia_Pintscher added a project: Wikidata-Sprint-2015-11-17.

TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, 
Joe, Wikidata-bugs, Mbch331, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-10 Thread daniel 
daniel added a comment.

This script show open one connection per db cluster, not one per wiki. If it 
does, that's a major bug. The change dispatcher makes use of the standard 
functionality provided by the LoadBalancer class, so the connections hsould be 
pooled. Maybe it fails to release the connection in some error case, leading to 
more error cases? I'll try look into it, but I'm not on my regular work 
schedule this week.

As to the performance issues:

- rewriting the dispatch logic to rely on the job queue has long been on the 
todo list, see T48643: [Story] Dispatching via delayed jobs (instead of cron 
script) . It's not trivial to get 
right, though.
- there is a proposal to significantly improve performance of the script, see 
T112245: [Task] Configure wikidata.org to rely on the wb_changes_subscription 
table for dispatching. 
- here's the epic for improving the dispatch process: T108944: [Epic] Improve 
change dispatching 

Regarding the PID lock: running one instance will not dispatch changes fast 
enough. The lag will build up.


TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, 
Joe, Wikidata-bugs, Mbch331, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-09 Thread Addshore 
Addshore edited projects, added MediaWiki-extensions-WikibaseRepository; 
removed Wikibase-Quality.
Addshore set Security to None.

TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, Aklapper, Joe, Wikidata-bugs, aude, Mbch331, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-09 Thread Addshore 
Addshore added a comment.

Related:

- https://phabricator.wikimedia.org/T111353
- https://phabricator.wikimedia.org/T108944
- https://phabricator.wikimedia.org/T48643
- https://phabricator.wikimedia.org/T70381
- https://phabricator.wikimedia.org/T70382


TASK DETAIL
  https://phabricator.wikimedia.org/T118162

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, Aklapper, Joe, Wikidata-bugs, aude, Mbch331, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs