Gehel added a comment.It seems that at the moment only .js and .css files which ar in the js / css directories are processed by filerev. Is there any reason to not process other files as well? It seems that we have .json files that could benefit from the same processing.TASK DETAILhttps
Gehel added a comment.We shoudl be able to also process at least:
vendor/jquery.uls/css/jquery.uls.css
logo.svg
They are unlikely to change frequently but still it would be nice...
At this point I'm not entirely sure which resources can be cached and which cannot. My guess:
main page
Gehel created this task.
Herald added subscribers: Zppix, Aklapper.
Herald added a project: Wikidata.
TASK DESCRIPTION
At the moment, WDQS exposes quite a few caching headers, some of them not
making sense. For example, ETags are generated by nginx and are different on
each nginx server
Gehel created this task.
Herald added subscribers: Zppix, Aklapper.
Herald added projects: Wikidata, Discovery.
TASK DESCRIPTION
As reported by Jasper Koehorst:
> I was updating the biological databases in wikidata when I encountered
cases where the official website was already ad
Gehel added a comment.
Doing a tcpdump capture, it seems that both request directly to nginx on
wdqs1001 and requests going through LVS + varnish (see the curl in above
comment) arrive correctly at Blazegraph. Blazegraph sends response in both
cases. Taking tcpdumps in front of nginx
Gehel placed this task up for grabs.
TASK DETAIL
https://phabricator.wikimedia.org/T127014
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: gerritbot, BBlack, Gehel, Nikki, Mbch331, Magnus, JanZerebecki, Smalyshev,
Aklapper, StudiesWorld
Gehel added a comment.
Placing this task up for grabs as it is unclear to me what will need to be
done once https://phabricator.wikimedia.org/T128813 is resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T127014
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Gehel claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T127014
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: BBlack, Gehel, Nikki, Mbch331, Magnus, JanZerebecki, Smalyshev, Aklapper,
StudiesWorld, Bugreporter, debt
Gehel moved this task to In progress on the
Discovery-Wikidata-Query-Service-Sprint workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T127014
WORKBOARD
https://phabricator.wikimedia.org/project/board/1239/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Gehel added a project: Discovery-Wikidata-Query-Service-Sprint.
TASK DETAIL
https://phabricator.wikimedia.org/T127014
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: BBlack, Gehel, Nikki, Mbch331, Magnus, JanZerebecki, Smalyshev, Aklapper
Gehel added a comment.
Continuing investigation:
1. capture tcpdump from cp1045: `sudo tcpdump -i any 'host
wdqs1001.eqiad.wmnet or port 80' -w varnish.pcap`. This should capture both
ingress and egress traffic to varnish.
2. send query directly to varnish from cp1045: `curl -k -D
Gehel created blocking task T133566: Reinstall and data reload of WDQS servers.
TASK DETAIL
https://phabricator.wikimedia.org/T123565
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Smalyshev, Gehel
Cc: Jheald, Ricordisamoa, Aklapper, Smalyshev
Gehel created this task.
TASK DESCRIPTION
As part of https://phabricator.wikimedia.org/T123565 we need to do a data
reload to re-index data for Geosearch. At the same time, we will do a full
reinstall of wdqs1001 to enable use of new disk space.
TASK DETAIL
https
Gehel added a comment.
General note: the grunt-hashres <https://gerrit.wikimedia.org/r/#/c/284428/>
change was meant mainly as an example of what I was talking about with Stas.
Grunt, Javascript and frontend dev in general are not really my cup of tea. The
important part for me is t
Gehel added a comment.
Nothing to do here on the WDQS side. Closing it.
TASK DETAIL
https://phabricator.wikimedia.org/T133046
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Smalyshev, Gehel
Cc: daniel, aude, IKhitron, Addshore, Aklapper
Gehel moved this task from Backlog to In progress on the
Discovery-Wikidata-Query-Service-Sprint board.
TASK DETAIL
https://phabricator.wikimedia.org/T133566
WORKBOARD
https://phabricator.wikimedia.org/project/board/1239/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Gehel added a project: Discovery-Wikidata-Query-Service-Sprint.
TASK DETAIL
https://phabricator.wikimedia.org/T133566
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: RobH, Cmjohnson, Stashbot, gerritbot, Smalyshev, Aklapper, Ricordisamoa
Gehel added a comment.
@JanZerebecki: could you add some more context to this task? It looks like
something that should be in my backlog, but I'm not entirely sure of what needs
doing. Looking at the prod nodes (wdqs1001 / wdqs1002), I see that all files
are owned by root (as they should
Gehel created this task.
TASK DESCRIPTION
During data reload on wdqs1002, wdqs-updater crashed. It was finally
restarted (most probably by systemd) and is now continuing the import.
Investigation is required to understand why this failed and if the data we have
imported is in a sufficiently
Gehel created blocking task T133986: Failure of wdqs-updater after data import.
TASK DETAIL
https://phabricator.wikimedia.org/T133566
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: RobH, Cmjohnson, Stashbot, gerritbot, Smalyshev, Aklapper
Gehel added a comment.
In https://phabricator.wikimedia.org/T133566#2236346, @Smalyshev wrote:
> Planned sequence:
>
> [X] (day before) Send email to the wikidata list
> [X] Take wdq1001 out of varnish config
> [ ] (IN PROGRESS) Shut down and reimage wdq10
Gehel added subscribers: Cmjohnson, RobH.
Gehel added a comment.
While rebuilding the RAID to add new disks, I realized wdqs1001 has 2x 300GB
+ 2x 150GB disks. I'm reinstalling anyway to ensure we don't run on a single
node, but it does not look like what was planned in
https
Gehel added a comment.
It seems that 300G disks were added to wdqs1002, but only 150G disks to
wdqs1001:
gehel@wdqs1001:~$ sudo megacli -PDList -aALL | grep "Raw Size"
Raw Size: 279.460 GB [0x22eec130 Sectors]
Raw Size: 279.460 GB [0x22eec130 Sectors]
Raw Size:
Gehel reopened blocking task T120712: install two Intel 320 Series
SSDSA2CW300G3 2.5" 300GB each in wdqs1001/wdqs1002 as "Open".
TASK DETAIL
https://phabricator.wikimedia.org/T119579
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To:
Gehel reopened this task as "Open".
Gehel set Security to None.
Herald added a subscriber: Southparkfan.
TASK DETAIL
https://phabricator.wikimedia.org/T120712
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Cmjohnson, Gehel
Cc: So
Gehel triaged this task as "High" priority.
Gehel claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: BBlack, Aklapper, Zppix, Lydia_Pintscher, Gehel, A
Gehel closed this task as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: BBlack, Aklapper, Zppix, Lydia_Pintscher, Gehel, Avner, debt, D3r1ck01,
FloNight, Iz
Gehel added a comment.
I can't reproduce the issue. Since it was random, I might just be lucky.
Closing this anyway for the moment, I'll reopen if I see the issue again.
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings
Gehel reopened this task as "Open".
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: BBlack, Aklapper, Zppix, Lydia_Pintscher, Gehel, Avner, debt, D3r1ck01,
FloNight, Iz
Gehel created this task.
Herald added subscribers: Zppix, Aklapper.
Herald added projects: Wikidata, Discovery.
TASK DESCRIPTION
At seemingly random intervals, WDQS responses are truncated after headers (no
body is sent). Example of the error:
$ curl -v
'https://query.wikidata.org
Gehel added a comment.
Seems the issue (or a similar one is still happening), I'm trying to
reproduce...
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: BBlack, Aklapper, Zppix
Gehel added a comment.
Replacement is scheduled for 1400 UTC, Thursday May 12.
TASK DETAIL
https://phabricator.wikimedia.org/T120712
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Cmjohnson, Gehel
Cc: Ricordisamoa, Gehel, Southparkfan, Cmjohnson
Gehel removed Gehel as the assignee of this task.
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: Stashbot, Luke081515, matmarex, TerraCodes, Urbanecm, KDDLB, hashar, Jonas,
gerritbot
Gehel added a comment.
Investigating:
- `May 03 09:33:48 wdqs1002 bash[13378]: java.io.IOException: Too many open
files`
- `May 03 09:33:48 wdqs1002 bash[13378]: at
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)`
- High number of TCP connections in TIME_WAIT / CLOSE
Gehel claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T134238
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: Aklapper, Yair_rand, Zppix, Avner, debt, Gehel, D3r1ck01, FloNight, Izno,
jkroll, Smalyshev, Wikidata-bugs
Gehel added a comment.
So the FD issue is probably a side issue that was detected because we were
looking more closely than usual. The main cause of this outage is most probably
a traffic peak as seen in Grafana
<https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?f
Gehel added a comment.
WDQS down again. 32K file handles taken by Java processes. More investigation
needed.
TASK DETAIL
https://phabricator.wikimedia.org/T134238
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: hoo, Frog23, Karima
Gehel added a comment.
CLOSE_WAIT is whan the TCP stack is waiting for the application to
acknowledge that the connection is terminated. So yes, a high number of
CLOSE_WAIT is an indication of a problem. If blazegraph is frozen (or Jetty
more precisely) that could explain. If I read
Gehel added a comment.
Random things I see (and have no idea if they are relevant to the issue):
- We have holes in Graphite data for unrelated metrics (for example load
average). Those holes do not completely coincide with blazegraph being
unavailable, but there seem to be some
Gehel added a comment.
Number of established connections on nginx is low and seem to all be coming
from varnish (I checked a sample over a few minutes):
gehel@wdqs1002:/etc/nginx$ sudo netstat -pn | grep 10.64.32.183:80 | grep
ESTABLISHED | wc -l
3
Number of connections
Gehel added a comment.
With updater shutdown, it seems that number of open pipes stays stable at
5-8K. I did not take those measurement long enough to be confident about the
correlation, but that's at least an idea. Of course, keeping the updater down
is not a solution...
TASK DETAIL
Gehel claimed this task.
Gehel added a project: Discovery-Wikidata-Query-Service-Sprint.
Gehel set Security to None.
TASK DETAIL
https://phabricator.wikimedia.org/T119915
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: gerritbot, Gehel
Gehel moved this task from Backlog to In progress on the
Discovery-Wikidata-Query-Service-Sprint board.
TASK DETAIL
https://phabricator.wikimedia.org/T119915
WORKBOARD
https://phabricator.wikimedia.org/project/board/1239/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Gehel added a comment.
Previous analysis is mostly wrong. File descriptors seem under control and
NOT jumping from 5k to 100k (if that was the case, I would be suspicious to
raise the limit that far).
TASK DETAIL
https://phabricator.wikimedia.org/T134238
EMAIL PREFERENCES
https
Gehel added a project: Discovery-Wikidata-Query-Service-Sprint.
TASK DETAIL
https://phabricator.wikimedia.org/T132387
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: Stashbot, JanZerebecki, gerritbot, MoritzMuehlenhoff, RobH, Smalyshev
Gehel moved this task to Done on the Discovery-Wikidata-Query-Service-Sprint
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T132387
WORKBOARD
https://phabricator.wikimedia.org/project/board/1239/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Gehel added a comment.
Herald added a subscriber: TerraCodes.
Side note: nginx computes etags based on last modified date and file size.
The last modified date might differ (not much, but enough) between our 2
servers, which means that we might have inconsistent etags. We should probably
Gehel added a comment.
This is related to https://phabricator.wikimedia.org/T133053.
TASK DETAIL
https://phabricator.wikimedia.org/T133046
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: Addshore, Aklapper, Smalyshev, Gehel, TerraCodes
Gehel created this task.
Herald added subscribers: TerraCodes, Aklapper.
Herald added a project: Wikidata.
TASK DESCRIPTION
The last update as seen in the logs of wdqs1001:
Apr 19 14:02:26 wdqs1001 bash[19132]: 14:02:26.456 [main] INFO
org.wikidata.query.rdf.tool.Update - Polled up
Gehel added a comment.
Might be related to RCStream breaking during the switch.
TASK DETAIL
https://phabricator.wikimedia.org/T133046
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: Aklapper, Smalyshev, Gehel, TerraCodes, Avner, debt
Gehel added a comment.
Looking at the code, it seems that updates are fetched by using the MW API.
Doing the call manually from wdqs1001 seems to return meaningful data:
curl -v -s
https://www.wikidata.org/w/api.php?format=json\=query\=recentchanges\=newer\=title\|ids\|timestamp\=100
Gehel added a comment.
In https://phabricator.wikimedia.org/T133026#2218744, @Smalyshev wrote:
> I checked the dates on both services and they are within 1s of each other,
so time difference should not be an issue.
As the etag is based on mtime, if mtime are different, et
Gehel added a comment.
grunt-hashres <https://github.com/Luismahou/grunt-hashres> or something
similar could be used to transform HTML before plublication and ensure that
URLs change whenever content changes. (note: I have no personal experience with
grunt-hashres).
TASK DETAIL
Gehel added a comment.
In https://phabricator.wikimedia.org/T133026#2221842, @daniel wrote:
> @Smalyshev yes, that's what I suspected. But that's not how load balancing
for static resources should work. It screws with the logic of IFM headers.
Varnish should hide the differen
Gehel added a comment.
@daniel the caching header configuration is well under my area of expertise.
All those are set in nginx conf
<https://github.com/wikimedia/operations-puppet/blob/production/modules/wdqs/templates/nginx.erb>.
So as soon as we have content based URLs I can tak
Gehel added a comment.
Strange ... Having a quick look at the HTTP responses and our nginx
configuration, it seems that the only resource on which we set caching headers
are the queries themselves (everything under `/bigdata/namespace/wdq/sparql`),
and even then we send a max-age=60. We do
Gehel added a blocked task: T133772: Rack and Setup new elastic search.
TASK DETAIL
https://phabricator.wikimedia.org/T120712
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Cmjohnson, Gehel
Cc: Gehel, Southparkfan, Cmjohnson, Smalyshev, StudiesWorld
Gehel added a blocked task: T133566: Reinstall and data reload of WDQS servers.
TASK DETAIL
https://phabricator.wikimedia.org/T120712
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Cmjohnson, Gehel
Cc: Gehel, Southparkfan, Cmjohnson, Smalyshev
Gehel removed a blocked task: T133772: Rack and Setup new elastic search.
TASK DETAIL
https://phabricator.wikimedia.org/T120712
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Cmjohnson, Gehel
Cc: Gehel, Southparkfan, Cmjohnson, Smalyshev
Gehel added a blocking task: T120712: install two Intel 320 Series
SSDSA2CW300G3 2.5" 300GB each in wdqs1001/wdqs1002.
TASK DETAIL
https://phabricator.wikimedia.org/T133566
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: Envlh,
Gehel added a comment.
Some discussion with Stas to clarify the need:
We do not declare a service in puppet, so puppet will not restart that service. This is on purpose, as we don't want updater to start before the initial import is done. This might be improved at some point.
As in this case
Gehel changed the title from "Deploy WDQS node on codfw" to "Deploy WDQS nodes on codfw".Gehel edited the task description. (Show Details)
EDIT DETAILSIn order to achieve better datacenter redundancy, we need to set up a node for Wikidata Query Service cluster in codfw. It
Gehel edited the task description. (Show Details)
EDIT DETAILSIn order to provide more robust service, avoid single-DC point of failure, and improve query service responsiveness, we'd like to create at least ofadd 2 WDQS server in codfw (see T124862). For this we need a machine close to the specs
Gehel edited the task description. (Show Details)
EDIT DETAILSIn order to provide more robust service, avoid single-DC point of failure, and improve query service responsiveness, we'd like to add 2 WDQS server in codfw (see T124862). For this we need a machine close to the specs with wdqs1001
Gehel edited the task description. (Show Details)
EDIT DETAILS...* 64G96G RAM...TASK DETAILhttps://phabricator.wikimedia.org/T138637EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: hoo, RobH, Aklapper, Joe, Gehel, Zppix, Smalyshev, Avner, debt
Gehel created this task.Gehel added projects: Wikidata-Query-Service, Discovery.Herald added subscribers: Zppix, Aklapper.Herald added a project: Wikidata.
TASK DESCRIPTIONOn July 6 around 7 UTC a user started to mass edit Wikidata, generating a flood of updates to WDQS, geenrating a slowdown
Gehel closed this task as "Resolved".Gehel claimed this task.Gehel added a comment.
No objections were raised, this is merged.TASK DETAILhttps://phabricator.wikimedia.org/T138627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Dzahn,
Gehel closed subtask T138627: Enable WDQS admins to enable/disable mask/unmask updater service as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T116754EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Gehel, Aklapper, JanZerebecki,
Gehel added a comment.
Title updated to reflect additional discussion that happened in comments.TASK DETAILhttps://phabricator.wikimedia.org/T138627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Dzahn, akosiaris, gerritbot, Zppix, JanZerebecki
Gehel added a subscriber: akosiaris.Gehel added a comment.
As pointed out by @akosiaris, what is actually needed is mask/unmaks, not enable/disable that is needed here.TASK DETAILhttps://phabricator.wikimedia.org/T138627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
Gehel edited the task description. (Show Details)
EDIT DETAILS...[X] - receive system, asset tag, attach packing slip to parent task T149351
[X] - rack in a different rack than wdqs100[12]
[X] - network port description and vlan update
[X] - setup mgmt dns (hostname and asset tag) and production
Gehel added a comment.
Initial installation is complete, data import in progress...TASK DETAILhttps://phabricator.wikimedia.org/T152643EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, GehelCc: ops-monitoring-bot, gerritbot, Southparkfan, Smalyshev
Gehel closed this task as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T152643EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, GehelCc: Stashbot, ops-monitoring-bot, gerritbot, Southparkfan, Smalyshev, Aklapper, Deskana, mark
Gehel closed subtask T152643: rack/setup/install wdqs1003 as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T153513EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, GehelCc: Esc3300, Harmonia_Amanda, Nikki, Gehel, Aklapper,
Gehel closed subtask T152643: rack/setup/install wdqs1003 as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T124627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: greg, Southparkfan, hoo, Gehel, RobH, Deskana, Tfinc, Smalyshev
Gehel reopened this task as "Open".Gehel added a comment.
Re-opening, wdqs100[12] still need a reimage to match this new data path. This was on hold, waiting for the additional wdqs1003 server to be configured. Now that it is here, time to finish this migration.TASK D
Gehel reopened subtask T144536: Move data storage to /srv/wdqs/ on codfw WDQS nodes as "Open".
TASK DETAILhttps://phabricator.wikimedia.org/T144380EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: thcipriani, Stashbot, gerritbot, Gehel
Gehel closed this task as "Resolved".Gehel claimed this task.Gehel added a comment.
This is actually done for some time, we have 3 nodes in codfw, matching the 3 nodes in eqiad.TASK DETAILhttps://phabricator.wikimedia.org/T124627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/sett
Gehel closed subtask T152644: rack/setup/install wdqs2003 as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T124627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: greg, Southparkfan, hoo, Gehel, RobH, Deskana, Tfinc, Smalyshev
Gehel closed this task as "Resolved".Gehel added a comment.
wdqs2003 has completed data import, it is now pooledTASK DETAILhttps://phabricator.wikimedia.org/T152644EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: ops-monitoring-bot,
Gehel created this task.Gehel added projects: Wikidata-Query-Service, Discovery-Search.Herald added a subscriber: Aklapper.Herald added projects: Wikidata, Discovery.
TASK DESCRIPTIONIn the last few days, we've have 3 instances of wdqs-updater lagging behind on some nodes (but not all). It looks
Gehel added a comment.
wdqs1001 has catched up on updates and is now repooled.TASK DETAILhttps://phabricator.wikimedia.org/T144536EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: ops-monitoring-bot, Stashbot, Aklapper, Gehel, gerritbot, Smalyshev
Gehel created this task.Gehel added a project: Wikidata-Query-Service.Herald added a subscriber: Aklapper.Herald added projects: Wikidata, Discovery.
TASK DESCRIPTIONSome LDF queries on wdqs1003 fail with:
java.lang.NoClassDefFoundError: Could not initialize class
Gehel edited the task description. (Show Details)
EDIT DETAILS...* add a call to [[ http://jhades.github.io/ | jhades ]] at application startupbuild time to check for stranger classpath issues duplicationTASK DETAILhttps://phabricator.wikimedia.org/T157703EMAIL PREFERENCEShttps
Gehel created this task.Gehel added projects: Wikidata-Query-Service, Discovery-Search (Current work).Herald added a subscriber: Aklapper.Herald added projects: Wikidata, Discovery.
TASK DESCRIPTIONToday, WDQS became unresponsive, leading to HTTP 502 errors. wdqs1003 was the first one to expose
Gehel triaged this task as "High" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T159248EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Aklapper, Gehel, Th3d3v1ls, EBjune, merbst, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xml
Gehel created this task.Gehel added projects: Operations, Wikidata-Query-Service, Discovery-Search.Herald added a subscriber: Aklapper.Herald added projects: Wikidata, Discovery.
TASK DESCRIPTIONWe only collect free / max / total memory on Blazegraph. It might make sense to also collect the GC
Gehel closed this task as "Resolved".Gehel claimed this task.Gehel added a comment.
New caching headers deployed. Checked with chrome, cache-control headers are showing up.TASK DETAILhttps://phabricator.wikimedia.org/T137238EMAIL PREFERENCEShttps://phabricator.wikimedia.org/sett
Gehel added a comment.
The cleanup of rules.log file is low priority and tracked on T144539. It will not be done as part of this task.TASK DETAILhttps://phabricator.wikimedia.org/T144380EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: gerritbot, Gehel
Gehel added a comment.
All known work is done and merged. Data import is in progress on codfw wdqs cluster which will validate that all this actually work. Steps left to be done:
data import on codfw (and check that everything work)
reinstall of eqiad cluster with new data directory
removal
Gehel created this task.Gehel added a project: Wikidata-Query-Service.Herald added a subscriber: Aklapper.Herald added projects: Wikidata, Discovery.
TASK DESCRIPTIONwdqs-updater is failing since Sept 7 ~13:30 UTC. The error seems related
Gehel created this task.Gehel added projects: Wikidata-Query-Service, Operations, Discovery.Herald added a subscriber: Aklapper.Herald added a project: Wikidata.
TASK DESCRIPTION@hoo noticed that some icinga checks on WDQS do not send notifications to wdqs-admins. A first attempt to fix
Gehel created subtask T144380: Install and configure new WDQS nodes on codfw.
TASK DETAILhttps://phabricator.wikimedia.org/T124862EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: hoo, Gehel, Tfinc, Deskana, Joe, EBernhardson, Aklapper, Smalyshev
Gehel created this task.Gehel added projects: Wikidata-Query-Service, Operations, Discovery.Herald added a subscriber: Aklapper.Herald added a project: Wikidata.
TASK DESCRIPTIONWe have 2 servers ready and available in codfw for WDQS (T142864). Configuration must now take place.
I want to take
Gehel added a project: Discovery-Wikidata-Query-Service-Sprint.Gehel added a comment.
Ideally, I'd like to have the data directory entirely configurable from puppet. At the moment, this is managed through symlinks declared both in puppet and in scap configuration. We should be able to simplify
Gehel added a comment.
The RWStore.properties is not actually created during scap deployment (we get the default one). There is some issue with the scap groups. I need to dig a bit more on how scap works...TASK DETAILhttps://phabricator.wikimedia.org/T144380EMAIL PREFERENCEShttps
Gehel added a comment.
I'm starting to understand a few things about scap:
the [codfw.wmnet] section in scap.cfg is activated if we deploy from codfw, not to codfw
--environment cannot be used to enable a specific section
It might just be easier to deploy a templated RWStore.properties on both
Gehel added a project: Discovery-Wikidata-Query-Service-Sprint.
TASK DETAILhttps://phabricator.wikimedia.org/T144948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Smalyshev, Gehel, Aklapper, hoo, mschwarzer, Avner, debt, D3r1ck01, Jonas, FloNight
Gehel added a subscriber: thcipriani.Gehel added a comment.
In T144380#2623579, @Smalyshev wrote:
the [codfw.wmnet] section in scap.cfg is activated if we deploy from codfw, not to codfw
That looks like a bug. Who needs different behavior on the same target depending on deployment host? Should
Gehel closed this task as "Resolved".Gehel added a comment.
the appropriate contact group has been added to the WDQS lag check.TASK DETAILhttps://phabricator.wikimedia.org/T144948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: gerritbot,
1 - 100 of 2396 matches
Mail list logo
