[Wikidata-bugs] [Maniphest] [Changed CC] T77897: Test host for Titan

2014-12-11 Thread Joe
Joe added a subscriber: Joe. TASK DETAIL https://phabricator.wikimedia.org/T77897 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc

[Wikidata-bugs] [Maniphest] [Changed CC] T77897: Test host for Titan

2014-12-11 Thread Joe
Joe removed a subscriber: Joe. TASK DETAIL https://phabricator.wikimedia.org/T77897 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe

[Wikidata-bugs] [Maniphest] [Changed CC] T77897: Test host for Titan

2014-12-11 Thread Joe
Joe added a subscriber: Joe. TASK DETAIL https://phabricator.wikimedia.org/T77897 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc

[Wikidata-bugs] [Maniphest] [Claimed] T77897: Test host for Titan

2014-12-11 Thread Joe
Joe claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T77897 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc

[Wikidata-bugs] [Maniphest] [Created] T78267: Package opendjk-8 for trusty

2014-12-11 Thread Joe
Joe created this task. Joe added a subscriber: Joe. Joe added a project: wikidata-query-service. Joe changed Security from none to none. TASK DESCRIPTION This is needed for the latest and shiniest version of Titan. So, we need to package this at high-priority. TASK DETAIL https

[Wikidata-bugs] [Maniphest] [Closed] T77897: Test host for Titan

2014-12-17 Thread Joe
Joe closed this task as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T77897 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] [Triaged] T77897: Test host for Titan

2014-12-18 Thread Joe
Joe triaged this task as Normal priority. TASK DETAIL https://phabricator.wikimedia.org/T77897 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] [Updated] T84902: deploy haedus and capella with os for orientdb testing

2014-12-21 Thread Joe
Joe added a project: wikidata-query-service. TASK DETAIL https://phabricator.wikimedia.org/T84902 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Commented On] T84902: deploy haedus and capella with os for orientdb testing

2014-12-21 Thread Joe
Joe added a comment. Haedus was NOT online post-install, it was in a busybox and it's presently reimaging because it could not find the root filesystem. TASK DETAIL https://phabricator.wikimedia.org/T84902 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim

[Wikidata-bugs] [Maniphest] [Commented On] T84902: deploy haedus and capella with os for orientdb testing

2014-12-22 Thread Joe
Joe added a comment. Capella only sees one disk, at pci-:01:00.0-scsi-0:2:0:0, this needs investigation. Moving the ticket to the codfw queue TASK DETAIL https://phabricator.wikimedia.org/T84902 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe

[Wikidata-bugs] [Maniphest] [Up For Grabs] T84902: deploy haedus and capella with os for orientdb testing

2014-12-22 Thread Joe
Joe placed this task up for grabs. Joe added a project: ops-codfw. TASK DETAIL https://phabricator.wikimedia.org/T84902 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Claimed] T78267: Package opendjk-8 for trusty

2014-12-22 Thread Joe
Joe claimed this task. Joe added projects: operations, ops-core. TASK DETAIL https://phabricator.wikimedia.org/T78267 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings

[Wikidata-bugs] [Maniphest] [Commented On] T78267: Package opendjk-8 for trusty

2014-12-22 Thread Joe
Joe added a comment. I created the package; I will not upload it to reprepro for now - I want to test it, on Titan maybe? TASK DETAIL https://phabricator.wikimedia.org/T78267 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username

[Wikidata-bugs] [Maniphest] [Updated] T75813: Investigate OrientDB for WDQ

2014-12-22 Thread Joe
Joe added a project: wikidata-query-service. TASK DETAIL https://phabricator.wikimedia.org/T75813 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Closed] T78267: Package opendjk-8 for trusty

2014-12-22 Thread Joe
Joe closed this task as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T78267 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] [Changed CC] T85513: Restoring Titan functionality after Cassandra disconnect

2015-01-05 Thread Joe
Joe added a subscriber: Joe. TASK DETAIL https://phabricator.wikimedia.org/T85513 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] [Claimed] T86561: Wikidata Query Service hardware

2015-01-12 Thread Joe
Joe claimed this task. Joe set Security to none. TASK DETAIL https://phabricator.wikimedia.org/T86561 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Commented On] T86561: Wikidata Query Service hardware

2015-02-09 Thread Joe
Joe added a comment. Yes, I was waiting for this evening's WQS meeting before reassessing priority/status, but marking it stalled is fair. TASK DETAIL https://phabricator.wikimedia.org/T86561 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T93658: cron job to create RDF dumps

2015-03-30 Thread Joe
Joe added a subscriber: Joe. TASK DETAIL https://phabricator.wikimedia.org/T93658 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo

[Wikidata-bugs] [Maniphest] [Commented On] T90109: BlazeGraph Finalization: Zookeeper

2015-02-23 Thread Joe
Joe added a comment. First of all, sorry If I did not get back to you earlier. I don't like the idea of having a complex tool like zookeeper running just to ensure HA. This is actually pretty bad for me, but still not a blocker per se. We surely don't want to run Zookeeper in share

[Wikidata-bugs] [Maniphest] [Commented On] T90109: BlazeGraph Finalization: Zookeeper

2015-02-23 Thread Joe
Joe added a comment. @Beebs.systap to be more explicit, it's highly probable we won't use ZK as our distributed, consistent KV store of choice internally, so maintaining a separated ZK cluster for blazegraph HA only would be too much of an hassle, hence my desire to co-host it. I also thought

[Wikidata-bugs] [Maniphest] [Commented On] T97930: edits seem to be very slow

2015-05-03 Thread Joe
Joe added a comment. After a few back and forth, We're pretty sure the cause of the outage was due to the changes in the jobchron service on the jobrunners that were released on saturday via https://gerrit.wikimedia.org/r/#/c/208408/ when I correctly restarted the jobchron service (which

[Wikidata-bugs] [Maniphest] [Closed] T97930: edits seem to be very slow

2015-05-03 Thread Joe
Joe closed this task as Resolved. Joe claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T97930 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Commented On] T97930: edits seem to be very slow

2015-05-03 Thread Joe
Joe added a comment. Users report editing is fast again. TASK DETAIL https://phabricator.wikimedia.org/T97930 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Commented On] T97930: edits seem to be very slow

2015-05-03 Thread Joe
Joe added a subscriber: Joe. Joe added a comment. We have identified an issue with connections to Redis, probably due to the fact that the jobrunners are not working since last night. We're working on it now. TASK DETAIL https://phabricator.wikimedia.org/T97930 REPLY HANDLER ACTIONS Reply

[Wikidata-bugs] [Maniphest] [Commented On] T103296: Wikidata and Wikiversity logo 404ing on wikimedia.org

2015-06-22 Thread Joe
Joe added a comment. we still see the other logos because they've been cached, apparently. I'm not sure what changed here, to be honest. TASK DETAIL https://phabricator.wikimedia.org/T103296 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc

[Wikidata-bugs] [Maniphest] [Commented On] T103296: Wikidata and Wikiversity logo 404ing on wikimedia.org

2015-06-22 Thread Joe
Joe added a subscriber: Joe. Joe added a comment. Ok, it seems related to the www.wikimedia.org apache config, in fact directly on an appserver I get: $ curl -H 'Host: en.wikipedia.org' localhost/static/images/project-logos/enwikiversity.png -I HTTP/1.1 200 OK Date: Mon, 22 Jun 2015 15

[Wikidata-bugs] [Maniphest] [Commented On] T103296: Wikidata and Wikiversity logo 404ing on wikimedia.org

2015-06-22 Thread Joe
Joe added a comment. Found it: $ ls /srv/mediawiki/docroot/wwwportal w/ there is no link to the static directory there, it must have been removed in some deploy, maybe eons ago. We noticed now as the cache on those objects has expired in the meanwhile TASK DETAIL https

[Wikidata-bugs] [Maniphest] [Updated] T103296: Wikidata and Wikiversity logo 404ing on wikimedia.org

2015-06-22 Thread Joe
Joe added a project: Release-Engineering. TASK DETAIL https://phabricator.wikimedia.org/T103296 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Joe, Krenair, zhuyifei1999, WMDE-Fisch, Legoktm, Addshore, Aklapper, Wikidata-bugs, RobH, aude

[Wikidata-bugs] [Maniphest] [Reassigned] T107152: Change hardware RAID controller on wmf3543, wmf3544

2015-07-28 Thread Joe
Joe reassigned this task from Joe to Cmjohnson. TASK DETAIL https://phabricator.wikimedia.org/T107152 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, Joe Cc: Manybubbles, Smalyshev, Joe, Aklapper, JanZerebecki, mark, faidon, Matanya, RobH

[Wikidata-bugs] [Maniphest] [Updated] T107152: Change hardware RAID controller on wmf3543, wmf3544

2015-07-28 Thread Joe
Joe removed a project: Patch-For-Review. Joe set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T107152 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Manybubbles, Smalyshev, Joe, Aklapper, JanZerebecki, mark, faidon

[Wikidata-bugs] [Maniphest] [Unblock] T86561: Wikidata Query Service hardware

2015-07-31 Thread Joe
Joe closed blocking task T107152: Change hardware RAID controller on wmf3543, wmf3544 as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: gerritbot, RobH, Matanya, faidon, mark

[Wikidata-bugs] [Maniphest] [Commented On] T86561: Wikidata Query Service hardware

2015-07-31 Thread Joe
Joe added a comment. wdqs1001 installed just fine (after I figured out I needed at least one deploy for trebuchet to work). Installing wdqs1002 now, then I'll set up the LVS config. TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Closed] T107152: Change hardware RAID controller on wmf3543, wmf3544

2015-07-31 Thread Joe
Joe closed this task as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T107152 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, Joe Cc: ksmith, Manybubbles, Smalyshev, Joe, Aklapper, JanZerebecki, mark, faidon, Matanya, RobH

[Wikidata-bugs] [Maniphest] [Unblock] T85159: [EPIC] Deploy a Wikidata Query Service into production

2015-07-31 Thread Joe
Joe closed blocking task T86561: Wikidata Query Service hardware as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T85159 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Addshore, Laddo, bd808, MBlissett, Krenair, waldyrious

[Wikidata-bugs] [Maniphest] [Closed] T86561: Wikidata Query Service hardware

2015-07-31 Thread Joe
Joe closed this task as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: gerritbot, RobH, Matanya, faidon, mark, JanZerebecki, Aklapper, Joe, Smalyshev, Manybubbles, jkroll

[Wikidata-bugs] [Maniphest] [Commented On] T86561: Wikidata Query Service hardware

2015-07-31 Thread Joe
Joe added a comment. Both servers are up and running and the wdqs-blazegraph service is running and the banner page shows on port 80. I will finish this work (adding LVS and eventually varnish support). Stas and James can now log into the machine and play with it - it's still not exposed

[Wikidata-bugs] [Maniphest] [Commented On] T107152: Change hardware RAID controller on wmf3543, wmf3544

2015-07-29 Thread Joe
Joe added a comment. @cmjohnson let me know when this is done, so that I can proceed to install the servers. TASK DETAIL https://phabricator.wikimedia.org/T107152 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, Joe Cc: Manybubbles

[Wikidata-bugs] [Maniphest] [Commented On] T107602: Set up a public interface to the wikidata query service

2015-08-05 Thread Joe
Joe added a comment. I've took a bit of an alternative approach: - deploy behind misc-web, as query.wikidata.org - as logstash does, do not use lvs but varnish directly here. TASK DETAIL https://phabricator.wikimedia.org/T107602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings

[Wikidata-bugs] [Maniphest] [Commented On] T107601: Assign an LVS service to the wikidata query service

2015-08-05 Thread Joe
Joe added a comment. I currently decided not to use an LVS loadbalancer for this, since it's being put behind misc-web. An LVS service might be needed once we scale up the service and (maybe) move it behind text-lb instead. TASK DETAIL https://phabricator.wikimedia.org/T107601 EMAIL

[Wikidata-bugs] [Maniphest] [Unblock] T85159: [EPIC] Deploy a Wikidata Query Service into production

2015-08-05 Thread Joe
Joe closed blocking task T107601: Assign an LVS service to the wikidata query service as Declined. TASK DETAIL https://phabricator.wikimedia.org/T85159 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Addshore, Laddo, bd808, MBlissett

[Wikidata-bugs] [Maniphest] [Declined] T107601: Assign an LVS service to the wikidata query service

2015-08-05 Thread Joe
Joe closed this task as Declined. TASK DETAIL https://phabricator.wikimedia.org/T107601 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: BBlack, Smalyshev, Joe, daniel, RobLa-WMF, Aklapper, aude, JanZerebecki, JeroenDeDauw, MrStradivarius

[Wikidata-bugs] [Maniphest] [Commented On] T107602: Set up a public interface to the wikidata query service

2015-08-03 Thread Joe
Joe added a comment. Stripping cookies at the varnish layer is possible, not adviceable in general IMO. TASK DETAIL https://phabricator.wikimedia.org/T107602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Legoktm, gerritbot, Smalyshev

[Wikidata-bugs] [Maniphest] [Claimed] T107602: Set up a public interface to the wikidata query service

2015-08-03 Thread Joe
Joe claimed this task. Joe set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T107602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Legoktm, gerritbot, Smalyshev, BBlack, Joe, daniel, RobLa-WMF, Aklapper, aude

[Wikidata-bugs] [Maniphest] [Commented On] T107601: Assign an LVS service to the wikidata query service

2015-08-03 Thread Joe
Joe added a comment. @Smalyshev the load balancer will depool one server if it goes down in after a very short interval. We're working on a tool to pool/depool servers in a timely fashion. TASK DETAIL https://phabricator.wikimedia.org/T107601 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] [Commented On] T86561: Wikidata Query Service hardware

2015-07-31 Thread Joe
Joe added a comment. I am currently installing wdqs1001; upon validation of the install, I'll add wdqs1002 as well. TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: gerritbot, RobH

[Wikidata-bugs] [Maniphest] [Commented On] T90115: BlazeGraph Security Review

2015-08-06 Thread Joe
Joe added a comment. @MoritzMuehlenhoff kernel upgrades should be doable relatively easily. If we want to be stricter about what the blazegraph user can do on the system, we can look into it too. I don't think network segregation is what we need here either. Alas, I don't see a general way

[Wikidata-bugs] [Maniphest] [Unblock] T85159: [EPIC] Deploy a Wikidata Query Service into production

2015-08-13 Thread Joe
Joe closed blocking task T107602: Set up a public interface to the wikidata query service as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T85159 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Addshore, Laddo, bd808, MBlissett

[Wikidata-bugs] [Maniphest] [Commented On] T107602: Set up a public interface to the wikidata query service

2015-08-13 Thread Joe
Joe added a comment. https://query.wikidata.org :) TASK DETAIL https://phabricator.wikimedia.org/T107602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: jeremyb, Yurik, JohnLewis, hoo, GWicke, greg, Lydia_Pintscher, csteipp, jcrespo

[Wikidata-bugs] [Maniphest] [Closed] T107602: Set up a public interface to the wikidata query service

2015-08-13 Thread Joe
Joe closed this task as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T107602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: jeremyb, Yurik, JohnLewis, hoo, GWicke, greg, Lydia_Pintscher, csteipp, jcrespo, Legoktm, gerritbot

[Wikidata-bugs] [Maniphest] [Commented On] T90115: BlazeGraph Security Review

2015-08-08 Thread Joe
Joe added a comment. In https://phabricator.wikimedia.org/T90115#1514604, @Smalyshev wrote: It also seems useful to limit the number of parallel queries per IP. I think as it gets to the servers, we don't have much info on the originating IP, as we will be talking to Varnish frontends. So

[Wikidata-bugs] [Maniphest] [Unblock] T95679: Make a puppet role that sets up a query service and loads it

2015-07-27 Thread Joe
Joe closed blocking task T105185: Need deploy rights for Wikidata Query Service as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T95679 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: GLavagetto, fgiunchedi, Lydia_Pintscher

[Wikidata-bugs] [Maniphest] [Closed] T105185: Need deploy rights for Wikidata Query Service

2015-07-27 Thread Joe
Joe closed this task as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T105185 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Dzahn, gerritbot, Matanya, Aklapper, Westonnh, Jdouglas, Joe, Smalyshev, jkroll, Wikidata-bugs, RobH

[Wikidata-bugs] [Maniphest] [Commented On] T86561: Wikidata Query Service hardware

2015-07-20 Thread Joe
Joe added a comment. Imaging wmf3543 as wdqs1001 TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: RobH, Matanya, faidon, mark, JanZerebecki, Aklapper, Joe, Smalyshev, Manybubbles, jkroll

[Wikidata-bugs] [Maniphest] [Updated] T86561: Wikidata Query Service hardware

2015-07-20 Thread Joe
Joe added a blocking task: T106320: wmf3543 can't install from PXE. TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: gerritbot, RobH, Matanya, faidon, mark, JanZerebecki, Aklapper, Joe

[Wikidata-bugs] [Maniphest] [Updated] T86561: Wikidata Query Service hardware

2015-07-15 Thread Joe
Joe removed a project: procurement. TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: RobH, Matanya, faidon, mark, JanZerebecki, Aklapper, Joe, Smalyshev, Manybubbles, jkroll, Wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections

2015-11-09 Thread Joe
Joe created this task. Joe added a subscriber: Joe. Joe added projects: operations, Database, Wikibase-Quality. Herald added a subscriber: Aklapper. Herald added a project: Wikidata. TASK DESCRIPTION I was looking at the logs for running maintenance scripts on the terbium replacement machine

[Wikidata-bugs] [Maniphest] [Unblock] T118363: widar oauth broken - magical script elves are temporarily ill

2015-11-11 Thread Joe
Joe closed blocking task T118372: OAuth broken as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T118363 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Sjoerddebruin, yuvipanda, Magnus, Harmonia_Amanda, Aklap

[Wikidata-bugs] [Maniphest] [Unblock] T118363: widar oauth broken - magical script elves are temporarily ill

2015-11-11 Thread Joe
Joe closed blocking task T118372: OAuth broken as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T118363 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Sjoerddebruin, yuvipanda, Magnus, Harmonia_Amanda, Aklap

[Wikidata-bugs] [Maniphest] [Unblock] T118363: widar oauth broken - magical script elves are temporarily ill

2015-11-11 Thread Joe
Joe closed blocking task T118372: OAuth broken as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T118363 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Sjoerddebruin, yuvipanda, Magnus, Harmonia_Amanda, Aklap

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-11-04 Thread Joe
Joe added a comment. @mobrovac so let me get this straight, we discussed something that was already overridden by an existing implementation? As far as deploying the python app, who is working on it? I think I can help with deployment/development of the server glue. TASK DETAIL https

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-06 Thread Joe
Joe added a comment. In https://phabricator.wikimedia.org/T114443#1703097, @Eevans wrote: > In https://phabricator.wikimedia.org/T114443#1701296, @Joe wrote: > > > Apart from the concerns on a practical use case which I agree with, I have > > a big doubt about the implementati

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-06 Thread Joe
Joe added a comment. So, doing some research on the topic, there is already a kafka rest proxy builtin into Kafka: http://docs.confluent.io/1.0/kafka-rest/docs/intro.html did you take a look at how it works? If we build this service, it could be a good idea to mimic its API I guess - at least

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-06 Thread Joe
Joe added a comment. In https://phabricator.wikimedia.org/T114443#1705509, @Ottomata wrote: > @Joe, there are two parts to this MVP: > > - Centralized (and CI controlled) schema sharing > - An easy way to get valid data into Kafka. > > With eventlogging right now, we

[Wikidata-bugs] [Maniphest] [Commented On] T104879: Define the details of the hardware we need to run WDQS

2015-07-08 Thread Joe
Joe added a comment. Just as a note - labs instances are incredibly slower than production hardware. IOPS are horrible, in particular, so if we just need 3x as an IO performance, we should be able to pull it off with non-ssd disks as well. TASK DETAIL https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Claimed] T105185: Need deploy rights for Wikidata Query Service

2015-07-08 Thread Joe
Joe claimed this task. Joe set Security to None. Herald added a subscriber: Matanya. TASK DETAIL https://phabricator.wikimedia.org/T105185 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Matanya, Aklapper, Westonnh, tomasz, Jdouglas, Joe

[Wikidata-bugs] [Maniphest] [Updated] T105185: Need deploy rights for Wikidata Query Service

2015-07-08 Thread Joe
Joe added a blocked task: T95679: Make a puppet role that sets up a query service and loads it. TASK DETAIL https://phabricator.wikimedia.org/T105185 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Matanya, Aklapper, Westonnh, tomasz

[Wikidata-bugs] [Maniphest] [Commented On] T105185: Need deploy rights for Wikidata Query Service

2015-07-09 Thread Joe
Joe added a comment. To clarify a bit, what will be needed is: - creating a wdqs-admins group in the admin module - grant access to tin (maybe this is optional) - grant access to the wdqs cluster once it exists :) - grant sudo rights to read logs and start/stop/restart wdqs TASK DETAIL https

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-04 Thread Joe
Joe added a comment. Apart from the concerns on a practical use case which I agree with, I have a big doubt about the implementation idea: I am in general a fan of the paradigm that it's better to beg for forgiveness than to ask for permission, and of Postel's robustness principle, so I don't

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-04 Thread Joe
Joe added a comment. In https://phabricator.wikimedia.org/T114443#1698223, @GWicke wrote: > @ottomata, main reason would be the ability to work with $simple_queue, > $binary_kafka, $amazon_queue and so on without changes in MW code. This isn't > so theoretical. We'll want a light

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T114443: EventBus MVP

2015-10-02 Thread Joe
Joe added a subscriber: Joe. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, Eevans, Ottomata, Matanya

[Wikidata-bugs] [Maniphest] [Closed] T105749: Need a way to create a systemd service that is initially stopped

2015-10-05 Thread Joe
Joe closed this task as "Resolved". Joe set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T105749 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Aklapper, Matanya, Smalyshev, jkroll, Wikidata-bugs, Jdouglas,

[Wikidata-bugs] [Maniphest] [Commented On] T105749: Need a way to create a systemd service that is initially stopped

2015-10-05 Thread Joe
Joe added a comment. For the record, this is possible by doing the following: base::service_unit{ 'foo': ensure

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-06 Thread Joe
Joe added a comment. Speaking with @ottomata, it seems that, contrarily to what I understood, this service would proxy connection to kafka only from producers and not from consumers. If this is the case, my concerns about reliability and performance are way less important of course. I am

[Wikidata-bugs] [Maniphest] [Commented On] T119579: Additional diskspace of wdqs1001/wdqs1002

2015-12-02 Thread Joe
Joe added a comment. btw, it should be noted that the LVM on those disks does not cover the whole free space as of now, so you still have room to grow; Also, we could add two new disks in raid1 and add them to the volume group of the LVM if we want to, without need for reinstalling the server

[Wikidata-bugs] [Maniphest] [Commented On] T119532: [Bug] Redirect for the www.wikidata.org/entity/ path should point to an HTTPS URL

2016-01-13 Thread Joe
Joe added a subscriber: Joe. Joe added a comment. @JanZerebecki does apache still need to be restarted? By the way, please note that wikitech instructions on apache changes https://wikitech.wikimedia.org/wiki/Application_servers state it pretty clearly that a manual restart is needed. I'll

[Wikidata-bugs] [Maniphest] [Updated] T135485: File deletion problem on commons.wikimedia.org

2016-05-18 Thread Joe
Joe removed a project: Operations. TASK DETAIL https://phabricator.wikimedia.org/T135485 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe Cc: Krenair, Storkk, aude, hoo, daniel, Pokefan95, Aklapper, Steinsplitter, Wdwd, Zppix, Riley_Huntley

[Wikidata-bugs] [Maniphest] [Commented On] T134238: Query service fails with "Too many open files"

2016-05-03 Thread Joe
Joe added a comment. @gehel what about unsetting the file descriptor limit for wdqs? If those are due to real use and not fd leaking, I mean. TASK DETAIL https://phabricator.wikimedia.org/T134238 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] [Updated] T142944: Performance and caching considerations for article placeholders accesses

2016-08-16 Thread Joe
Joe added a project: Traffic. TASK DETAILhttps://phabricator.wikimedia.org/T142944EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: JoeCc: daniel, Lydia_Pintscher, Joe, Lucie, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, faidon

[Wikidata-bugs] [Maniphest] [Commented On] T159618: Job queue rising to nearly 3 million jobs

2017-03-06 Thread Joe
Joe added a comment. @Lydia_Pintscher not really, I'm monitoring the jobqueue and it's constantly decreasing in size. We should be ok.TASK DETAILhttps://phabricator.wikimedia.org/T159618EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: JoeCc: Lydia_Pintscher

[Wikidata-bugs] [Maniphest] [Commented On] T171928: Wikidata database locked

2017-07-28 Thread Joe
Joe added a comment. I just did two test edits, I can confirm it works.TASK DETAILhttps://phabricator.wikimedia.org/T171928EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: JoeCc: Joe, jcrespo, greg, Mbch331, Smalyshev, MisterSynergy, TerraCodes, Jay8g

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-09-19 Thread Joe
Joe added a comment. FWIW we're seeing another almost-incontrollable growth of jobs on commons and probably other wikis. I might decide to raise the concurrency of those jobs.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-09-07 Thread Joe
Joe added a comment. I did some more number crunching on the instances of runJob.php I'm running on terbium, I found what follows: Wikibase refreshlinks jobs might benefit from being in smaller batches, as many of those are taking a long time to execute. Out of 33.4k wikibase jobs, we had

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-09-06 Thread Joe
Joe added a comment. In T173710#3581849, @aaron wrote: Those refreshLInks jobs (from wikibase) are the only ones that use multiple titles per job, so they will be a lot slower (seems to be 50 pages/job) than the regular ones from MediaWiki core. That is a bit on the slow side for a run time

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-09-05 Thread Joe
Joe added a comment. We still have around 1.4 million items in queue for commons, evenly divided between htmlCacheUpdate jobs and refreshLinks jobs. I've started a few runs of the refreshLinks job and since yesterday most jobs are just processing the same root job from August 26th. Those jobs

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-09-06 Thread Joe
Joe added a comment. In T173710#3584505, @Krinkle wrote: In T173710#3583445, @Joe wrote: As a side comment: this is one of the cases where I would've loved to have an elastic environment to run MediaWiki-related applications: I could've spun up 10 instances of jobrunner dedicated to refreshlinks

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-09-29 Thread Joe
Joe added a comment. oblivian@terbium:~$ /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/group1.dblist showJobs.php --group | awk '{if ($3 > 1) print $_}' cawiki: refreshLinks: 104355 queued; 3 claimed (3 active, 0 abandoned); 0 delayed commonswiki: refreshLinks: 2073193 queued;

[Wikidata-bugs] [Maniphest] [Commented On] T176312: Don’t check format constraint via SPARQL (safely evaluating user-provided regular expressions)

2017-09-25 Thread Joe
Joe added a comment. I think re2 seems like an interesting candidate. I would argue we still want to have a separate microservice running on a separate cluster from MediaWiki, for security reasons, and I would think it could be used to run the regular _expression_ validations as well. AIUI

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-08-31 Thread Joe
Joe added a comment. @aaron so you're saying that when we have someone editing a lot of pages with a lot of backlinks we will see the jobqueue growing basically for quite a long time, as the divided jobs will be executed at a later time, and as long as the queue is long enough, we'll see jobs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T173710: Job queue is increasing non-stop

2017-08-31 Thread Joe
Joe added a subscriber: ema.Joe added a comment. Correcting myself after a discussion with @ema: since we have up to 4 cache layers (at most), we should process any job with a root timestamp newer than 4 times the cache TTL cap. So anything older than 4 days should be safely discardable

[Wikidata-bugs] [Maniphest] [Updated] T178810: Wikibase: Increase batch size for HTMLCacheUpdateJobs triggered by repo changes.

2017-10-23 Thread Joe
Joe added a project: User-Joe. TASK DETAILhttps://phabricator.wikimedia.org/T178810EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: JoeCc: Aklapper, aude, thiemowmde, hoo, Ladsgroup, Krinkle, Joe, daniel, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Zppix

[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken

2018-02-28 Thread Joe
Joe added a comment. In T188045#4007098, @Smalyshev wrote: I wonder if it's possible to use one of the new servers we're getting in T187766 to restore full capacity if debugging what is going on with 1004 takes time. Would it be a good thing to do? If losing one server out of 4 is an issue

[Wikidata-bugs] [Maniphest] [Commented On] T205865: Investigate decrease in wikidata dispatch times due to eqiad -> codfw DC switch

2018-10-01 Thread Joe
Joe added a comment. Have you checked if the latest changes didn't just switch execution from HHVM to PHP7? That could explain a better performance. Also, can I ask which redis servers are interacted with? I guess the ones for the locking system, right?TASK DETAILhttps://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Commented On] T205865: Investigate decrease in wikidata dispatch times due to eqiad -> codfw DC switch

2018-10-01 Thread Joe
Joe added a comment. In T205865#4630573, @Joe wrote: Have you checked if the latest changes didn't just switch execution from HHVM to PHP7? That could explain a better performance. I can answer this: no, they did not. We launch mwscript for the dispatcher with PHP='hhvm -vEval.Jit=1' Also, can

[Wikidata-bugs] [Maniphest] [Commented On] T213318: Wikibase Front-End Architecture

2019-01-16 Thread Joe
Joe added a comment. In T213318#4885332, @daniel wrote: So, in conclusion, Wikidata has a lot of edits, but several magnitudes fewer views than a Wikipedia of comparable size. So, while MediaWiki generally optimizes for heavy read loads, the Wikidata UI should be optimized for frequent edits

[Wikidata-bugs] [Maniphest] [Commented On] T213318: Wikibase Front-End Architecture

2019-01-20 Thread Joe
Joe added a comment. In T213318#4888367, @Nikerabbit wrote: Do I understand this correctly, that this would add a mandatory Nodejs service to run a Wikibase installation? Is there no client side rendering support planned initially? As an sysadmin for couple of third party wikis (some of which use

[Wikidata-bugs] [Maniphest] [Commented On] T213318: Wikibase Front-End Architecture

2019-01-15 Thread Joe
Joe added a comment. Moving (even part of) the presentation layer outside of MediaWiki raises quite a few questions we have to make important decisions about. But in the case of Wikidata, I can see how an exception could be made: It's not part of the core functionality of MediaWiki Its UI

[Wikidata-bugs] [Maniphest] [Commented On] T212189: New Service Request: Wikidata Termbox SSR

2018-12-18 Thread Joe
Joe added a comment. Looking at the attached diagrams, it seems that the flow of a request is as follows: page gets requested to MediaWiki MW sends a request to the rendering service the rendering service sends request(s) to mediawiki via api.php to fetch the data, and sends back the rendered

[Wikidata-bugs] [Maniphest] [Commented On] T212189: New Service Request: Wikidata Termbox SSR

2018-12-18 Thread Joe
Joe added a comment. Also: it is stated in https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/SSR_Service that "In case of no configured server-side rendering service or a malfunctioning of it, the client-side code will act as a fallback". This is a bit the other way around with respe

[Wikidata-bugs] [Maniphest] [Commented On] T212189: New Service Request: Wikidata Termbox SSR

2018-12-21 Thread Joe
Joe added a comment. Also, if we're going to build microservices, I'd like to not see applications that "grow", at least in terms of what they can do. A microservice should do one thing and do it well. In this case, it's using data from mediawiki to render an HTML fragment; unless you wa

  1   2   >