[Wikidata-bugs] [Maniphest] T298525: Tune "BlazegraphFreeAllocatorsDecreasingRapidly" alerts

bking renamed this task from "Tune "BlazegraphFreeAllocatorsDecreasingRapidly"" to "Tune "BlazegraphFreeAllocatorsDecreasingRapidly" alerts". TASK DETAIL https://phabricator.wikimedia.org/T298525 EMAIL PREFERENCES https://phabricator.wikimedia.org/set

[Wikidata-bugs] [Maniphest] T298525: Tune "BlazegraphFreeAllocatorsDecreasingRapidly" alerts

bking added a comment. Related commits here <https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+log/refs/heads/master/team-search-platform/blazegraph.yaml> TASK DETAIL https://phabricator.wikimedia.org/T298525 EMAIL PREFERENCES https://phabricator.wikimedia.org/se

[Wikidata-bugs] [Maniphest] T298525: Tune "BlazegraphFreeAllocatorsDecreasingRapidly"

bking added a subscriber: dcausse. bking added a comment. More context from @dcausse : The alert is managed by Alertmanager, code stored in Gerrit <https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-search-platform/blazegraph.yaml>

[Wikidata-bugs] [Maniphest] T296470: Initialize WCQS production servers

bking added a comment. Started data load via tmux session on cumin1001 at ~ `Tue Jan 11 16:53:46 2022` . Expected to take at least 24 hours. TASK DETAIL https://phabricator.wikimedia.org/T296470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

bking added a comment. Per messages above, we have completely failed over the wdqs and wdqs-internal services from eqiad to codfw. TASK DETAIL https://phabricator.wikimedia.org/T302494 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

bking added a comment. Manually installed on wdqs1010 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, bking Cc: bking, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot

[Wikidata-bugs] [Maniphest] T301953: Investigate wdqs1013 stability issues

bking added a comment. Suggestions: - Data reload - Server reimage - Hardware tests - Close observation over a limited time TASK DETAIL https://phabricator.wikimedia.org/T301953 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc

[Wikidata-bugs] [Maniphest] T301953: Investigate wdqs1013 stability issues

bking claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T301953 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: bking, Aklapper, Zbyszko, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz

[Wikidata-bugs] [Maniphest] T303134: Should wdqs LVS checks page

bking claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T303134 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: jbond, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. Another piece of the puzzle, some wdqs hosts use MDRAID for their /srv partition, some use LVM <https://phabricator.wikimedia.org/P23901> . Working assumption is that only the LVM hosts will take forever to reboot. TASK DETAIL https://phabricator.wikimed

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. Actions tried so far: disabling swap via systemd before rebooting. Worked on `wdqs2007`, did not work on `wdqs2002`. Also worth noting is that we had previously rebooted `wdqs2007` within the last 30 minutes, so a minor kernel update (from 4.19.0-16-amd64 to 4.19.0-20

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. This is still happening, @RKemper found some interesting links that could explain this behavior: https://wiki.freedesktop.org/www/Software/systemd/Debugging/#diagnosingshutdownproblems https://old.reddit.com/r/archlinux/comments/ba3zec

[Wikidata-bugs] [Maniphest] T305162: Determine if requestctl is appropriate for WDQS

bking created this task. bking added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Per @dcausse suggestion at today's retro, evaluate new SRE tool 'requestctl' . Could we use this to protect the WDQS service from abusive requests

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. Correction: both MDRAID and LVM servers have this problem. Both services' systemd unit files have the same "Conflicts=shutdown.target" directive. Still haven't tried the systemd workaround though, will test that today. TASK DETAIL https://phabricator.wik

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. Unfortunately, the systemd workaround listed above did **not** work. We will try adjusting some other unit file values when time permits. TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Paste] P23901: SWRAID vs LVM across wdqs prod

bking added a project: Wikidata-Query-Service. PASTE DETAIL https://phabricator.wikimedia.org/P23901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas

[Wikidata-bugs] [Maniphest] T242453: Detect and alert and/or remediate Blazegraph deadlocks

bking added a comment. Per conversation with dcausse, we could potentially run jstack on a timer and grep the output for errors as shown above, then alert and/or remediate. TASK DETAIL https://phabricator.wikimedia.org/T242453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings

[Wikidata-bugs] [Maniphest] T242453: Detect and alert and/or remediate Blazegraph deadlocks

bking renamed this task from "Deadlock in blazegraph blocking all queries and updates" to "Detect and alert and/or remediate Blazegraph deadlocks". TASK DETAIL https://phabricator.wikimedia.org/T242453 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/pa

[Wikidata-bugs] [Maniphest] T344882: Some servers for the Commons query service (WCQS) are missing data

bking moved this task from Ready for Work to Done on the Data-Platform-SRE board. bking added a comment. Hello, I've fixed wdqs1003 as well and it appears we have the same amount of triples now <https://grafana.wikimedia.org/d/00489/wikidata-query-service?orgId=1=1m=now-24h=now_n

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata, bking, Aklapper, dcausse, Danny_Benjafield_WMDE

[Wikidata-bugs] [Maniphest] T337013: [Epic] Splitting the graph in WDQS

bking added a subscriber: RKemper. bking added a comment. When T345475 <https://phabricator.wikimedia.org/T345475> is done, we should have 3 new WDQS hosts in CODFW that could be used for the graph splitting experiment. @RKemper let us know if you have any objections to this plan.

[Wikidata-bugs] [Maniphest] T337013: [Epic] Splitting the graph in WDQS

bking added a subtask: T345475: Service implementation for wdqs202[3-5].codfw.wmnet. TASK DETAIL https://phabricator.wikimedia.org/T337013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, bking, tfmorris, elal, karapayneWMDE

[Wikidata-bugs] [Maniphest] T346456: Improve concurrency limits configuration of the wdqs updater

bking added a parent task: T342149: Test common operations in the flink operator/k8s/Flink ZK environment. TASK DETAIL https://phabricator.wikimedia.org/T346456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking

[Wikidata-bugs] [Maniphest] T344882: Some servers for the Commons query service (WCQS) are missing data

bking added a comment. I've repooled wcqs1001 and wcqs1002 after verifying they have the correct amount of triples. We've left wcqs1003 depooled as we troubleshoot further. Let us know if you notice any other issues. TASK DETAIL https://phabricator.wikimedia.org/T344882 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T344882: Some servers for the Commons query service (WCQS) are missing data

bking added a comment. Thanks for bringing this to our attention, and sorry for the inconvenience. I have depooled the eqiad datacenter while we work to address the issue. Please test our your queries and let us know if you're still getting inconsistent results. TASK DETAIL https

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata, bking, Aklapper, dcausse, Danny_Benjafield_WMDE, fbalicchia

[Wikidata-bugs] [Maniphest] T337296: Allow federated queries with the NLG endpoint (data.nlg.gr)

bking added a comment. Thanks for the quick response. It could very well be our fault, as we just changed how we deploy our federation allowlist <https://phabricator.wikimedia.org/T343856> . Thus, we'll continue to troubleshoot from our side as well. TASK DETAIL

[Wikidata-bugs] [Maniphest] T337296: Allow federated queries with the NLG endpoint (data.nlg.gr)

bking added a comment. @Epidosis sorry for the delay on this ticket. We've added your endpoint, can you please test it and let us know if it works? Thanks for your patience. TASK DETAIL https://phabricator.wikimedia.org/T337296 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] T326914: Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink

bking added a project: Data-Platform-SRE. TASK DETAIL https://phabricator.wikimedia.org/T326914 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: bking, dcausse, Aklapper, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis

[Wikidata-bugs] [Maniphest] T344284: Rename usages of whitelist to allowlist in query service rdf repo

bking moved this task from Ready for Work to Done on the Data-Platform-SRE board. bking added a comment. We successfully deployed this yesterday; moving to "Done" on the workboard. TASK DETAIL https://phabricator.wikimedia.org/T344284 WORKBOARD https://phabricator.wikimedia.o

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a subtask: T349011: Improve data-reload cookbook based on graph split needs. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, dcausse, Aklapper, bking

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a comment. Apologies for not catching this earlier: - to be aligned with the dumps that are imported into hdfs we must select a particular set of files, when selecting a "-all" file the preceding "lexemes" one must be taken, e.g. if wikidata-20230

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking created this task. bking added projects: Wikidata-Query-Service, Data-Platform-SRE. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION We have tested the flink operator mode in dse-k8s <https://phabricator.wikimedia.org/T342149> . Our next step is to m

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata, bking, Aklapper, dcausse, Danny_Benjafield_WMDE

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking added a subtask: T349095: Migrate staging rdf-streaming-updater to flink operator. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a parent task: T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model. TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking added a subtask: T346456: Improve concurrency limits configuration of the wdqs updater. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata

[Wikidata-bugs] [Maniphest] T346456: Improve concurrency limits configuration of the wdqs updater

bking added a parent task: T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model. TASK DETAIL https://phabricator.wikimedia.org/T346456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, bking Cc

[Wikidata-bugs] [Maniphest] T346456: Improve concurrency limits configuration of the wdqs updater

bking removed a parent task: T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model. TASK DETAIL https://phabricator.wikimedia.org/T346456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, bking Cc

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking removed a subtask: T346456: Improve concurrency limits configuration of the wdqs updater. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a comment. Progress report: `wdqs1022`: started reload 2023-10-24 UTC . Munging finished 2023-10-26 0003 UTC. So far, we've processed 409/1104 munged files, which works out to ~37% complete over a period of ~1 wk total, ~5 days if we don't count the munging step. Assuming

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking closed subtask T349011: Improve data-reload cookbook based on graph split needs as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, dcausse, Aklapper, bking

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a comment. Current status: flink-operator is listening for rdf-streaming-updater rdf-streaming-updater job deploys, but it seems like it can't connect to kafka: {"@timestamp":"2023-11-06T23:03:13.111Z","log.level": "INFO

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a comment. In T347504#9307835 <https://phabricator.wikimedia.org/T347504#9307835>, @dcausse wrote: > @bking thanks for triggering the import, could you update the task description with the dump files you used? (needed because we have to explicitly keep the corr

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, dcausse, Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis

[Wikidata-bugs] [Maniphest] T349147: Follow up on rdf-streaming-updater failure 2023-10-17

bking closed this task as "Resolved". bking moved this task from In Progress to Done on the Data-Platform-SRE board. bking added a comment. Redeployed WCQS and WDQS jobs in eqiad and codfw envs: INFO 2023-10-23T18:20:23+ [ root] Job WDQS Streaming Updater saved at

[Wikidata-bugs] [Maniphest] T347605: Document process for getting JNL files/consider automation

bking triaged this task as "Low" priority. bking closed this task as "Resolved". bking added a comment. In T347605#9249978 <https://phabricator.wikimedia.org/T347605#9249978>, @dr0ptp4kt wrote: > @bking just wanted to express my gratitude for the support on

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata, bking, Aklapper, dcausse, Danny_Benjafield_WMDE

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata, bking, Aklapper, dcausse, Danny_Benjafield_WMDE

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking added a subtask: T342149: Test common operations in the flink operator/k8s/Flink ZK environment. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena

[Wikidata-bugs] [Maniphest] T347284: Restore service for https://query.wikidata.org/bigdata/ldf

bking renamed this task from "https://query.wikidata.org/bigdata/ldf is broken" to "Restore service for https://query.wikidata.org/bigdata/ldf;. TASK DETAIL https://phabricator.wikimedia.org/T347284 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/e

[Wikidata-bugs] [Maniphest] T344905: Publish WDQS JNL files to dumps.wikimedia.org

bking added a project: Data-Platform-SRE. TASK DETAIL https://phabricator.wikimedia.org/T344905 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: bking, Krinkle, dr0ptp4kt, Abbe98, Gehel, Addshore, Aklapper, Danny_Benjafield_WMDE, Mohamed

[Wikidata-bugs] [Maniphest] T347605: Document process for getting JNL files/consider automation

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347605 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Addshore, dr0ptp4kt, Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis

[Wikidata-bugs] [Maniphest] T347605: Document process for getting JNL files/consider automation

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347605 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Addshore, dr0ptp4kt, Aklapper, bking, AWesterinen, BTullis, Namenlos314, Gq86, Lucas_Werkmeister_WMDE

[Wikidata-bugs] [Maniphest] T347605: Document process for getting JNL files/consider automation

bking created this task. bking added projects: Wikidata-Query-Service, Data-Platform-SRE. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Blazegraph (the application that serves WDQS) stores all its data in a single JNL file. The WDQS file is very large (~1.2TB) so moving

[Wikidata-bugs] [Maniphest] T347605: Document process for getting JNL files/consider automation

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347605 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, AWesterinen, BTullis, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst

[Wikidata-bugs] [Maniphest] T344905: Publish WDQS JNL files to dumps.wikimedia.org

bking added a comment. @dr0ptp4kt and I were looking at this today and it occurred to me that the JNL file is uncompressed. Thus, I gzipped the main wikidata JNL file from `wdqs1016`, which takes ~4 hours using pigz at maximum compression rate, and we end up with a ~400 GB file

[Wikidata-bugs] [Maniphest] T347284: Restore service for https://query.wikidata.org/bigdata/ldf

bking added a project: Data-Platform-SRE. TASK DETAIL https://phabricator.wikimedia.org/T347284 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, bking, MisterSynergy, dcausse, Aklapper, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T347284: Restore service for https://query.wikidata.org/bigdata/ldf

bking closed this task as "Resolved". bking moved this task from Incoming to Done on the Data-Platform-SRE board. bking claimed this task. bking added a comment. @MisterSynergy After applying the last patch, https://query.wikidata.org/bigdata/ldf seems to be back up. Thus, I'm

[Wikidata-bugs] [Maniphest] T347355: Create alerts for https://query.wikidata.org/bigdata/ldf

bking created this task. bking added projects: Data-Platform-SRE, Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Per T347284 <https://phabricator.wikimedia.org/T347284> , we lost the LDF endpoint for a few days. Creating this ticket to add

[Wikidata-bugs] [Maniphest] T347355: Create alerts for https://query.wikidata.org/bigdata/ldf

bking added a parent task: T347284: Restore service for https://query.wikidata.org/bigdata/ldf. TASK DETAIL https://phabricator.wikimedia.org/T347355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, AWesterinen, BTullis

[Wikidata-bugs] [Maniphest] T347284: Restore service for https://query.wikidata.org/bigdata/ldf

bking added a subtask: T347355: Create alerts for https://query.wikidata.org/bigdata/ldf. TASK DETAIL https://phabricator.wikimedia.org/T347284 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, bking, MisterSynergy, dcausse

[Wikidata-bugs] [Maniphest] T347605: Document process for getting JNL files/consider automation

bking added a comment. A few notes on this process: - I used zstd compression <https://facebook.github.io/zstd/> to compress the JNL file, as it supposedly offers the best speed. I used `zstd -T0 -19 wikidata.jnl` as my compression command (all cores, maximum compression),

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a subtask: T347505: Prepare new WDQS hosts for graph splitting. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking added a parent task: T347504: WDQS graph split: load data from dumps into new hosts. TASK DETAIL https://phabricator.wikimedia.org/T347505 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, AWesterinen, BTullis

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking created this task. bking added projects: Data-Platform-SRE, Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION wdqs1017-1024 are newly-deployed WDQS hosts. Let's claim 2022-2024 for the graph splitting. These will be a new tier of WDQS

[Wikidata-bugs] [Maniphest] T337013: [Epic] Splitting the graph in WDQS

bking added a subtask: T347505: Prepare new WDQS hosts for graph splitting. TASK DETAIL https://phabricator.wikimedia.org/T337013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: dr0ptp4kt, RKemper, bking, tfmorris, elal, karapayneWMDE

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking added a parent task: T337013: [Epic] Splitting the graph in WDQS. TASK DETAIL https://phabricator.wikimedia.org/T347505 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, AWesterinen, BTullis, Namenlos314, Gq86

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a subscriber: dcausse. bking added a comment. @dcausse couple of questions: - Are we OK to start the data load as soon as these hosts are in production? - Does each host need its data loaded, or can we load on one and data-transfer to the others? TASK DETAIL https

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis, karapayneWMDE, Invadibot

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347505 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: dcausse, Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis, karapayneWMDE

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a parent task: T347505: Prepare new WDQS hosts for graph splitting. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking removed a subtask: T347505: Prepare new WDQS hosts for graph splitting. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking added a subtask: T347504: WDQS graph split: load data from dumps into new hosts. TASK DETAIL https://phabricator.wikimedia.org/T347505 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, AWesterinen, BTullis, Namenlos314

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking removed a parent task: T347504: WDQS graph split: load data from dumps into new hosts. TASK DETAIL https://phabricator.wikimedia.org/T347505 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, AWesterinen, BTullis

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a parent task: T337013: [Epic] Splitting the graph in WDQS. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper, bking, AWesterinen, BTullis, Namenlos314, Gq86

[Wikidata-bugs] [Maniphest] T337013: [Epic] Splitting the graph in WDQS

bking added a subtask: T347504: WDQS graph split: load data from dumps into new hosts. TASK DETAIL https://phabricator.wikimedia.org/T337013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: dr0ptp4kt, RKemper, bking, tfmorris, elal

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking created this task. bking added projects: Wikidata-Query-Service, Data-Platform-SRE. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION In order to proceed on T337013 <https://phabricator.wikimedia.org/T337013> , we need to do a full data reload on three hosts, s

[Wikidata-bugs] [Maniphest] T337013: [Epic] Splitting the graph in WDQS

bking removed a subtask: T345475: Service implementation for wdqs202[3-5].codfw.wmnet. TASK DETAIL https://phabricator.wikimedia.org/T337013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: dr0ptp4kt, RKemper, bking, tfmorris, elal

[Wikidata-bugs] [Maniphest] T326914: Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink

bking added a parent task: T342149: Test common operations in the flink operator/k8s/Flink ZK environment. TASK DETAIL https://phabricator.wikimedia.org/T326914 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, bking Cc: dr0ptp4kt, bking

[Wikidata-bugs] [Maniphest] T326914: Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink

bking added a parent task: T340548: [EPIC] Deployment of the Search Update Pipeline on Flink / k8s. TASK DETAIL https://phabricator.wikimedia.org/T326914 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, bking Cc: dr0ptp4kt, bking, dcausse

[Wikidata-bugs] [Maniphest] T347605: Document process for getting JNL files/consider automation

bking added a comment. Checksum does not match the version from `wdqs1016`, which is: sha1sum wikidata.jnl.zst e3197eb5177dcd1aa0956824cd8dc4afc2d8796c wikidata.jnl.zst I also downloaded the file locally after putting it up in Cloudflare, which has a different checksum

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, dcausse, Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking claimed this task. bking moved this task from Incoming to In Progress on the Data-Platform-SRE board. TASK DETAIL https://phabricator.wikimedia.org/T347504 WORKBOARD https://phabricator.wikimedia.org/project/board/6524/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347505 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: dcausse, Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis, karapayneWMDE

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a comment. More details related to dump loading in T325114 <https://phabricator.wikimedia.org/T325114> . TASK DETAIL https://phabricator.wikimedia.org/T347504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: RKemper, d

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T347505 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: dcausse, Aklapper, bking, Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, Jersione

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking closed subtask T341792: Provision Zookeeper Cluster for storing Flink HA data as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata, bking, Aklapper, dcausse, Danny_Benjafield_WMDE

[Wikidata-bugs] [Maniphest] T347504: WDQS graph split: load data from dumps into new hosts

bking added a comment. I started a data reload for hosts`wdqs1022-1024`. These are running in a tmux window under my user on`cumin1001`. Based on T323096 <https://phabricator.wikimedia.org/T323096> , we expect this process to fail multiple times, which is why we're running in on

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking reopened subtask T342149: Test common operations in the flink operator/k8s/Flink ZK environment as In Progress. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking closed subtask T342149: Test common operations in the flink operator/k8s/Flink ZK environment as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking changed the status of subtask T342149: Test common operations in the flink operator/k8s/Flink ZK environment from Open to In Progress. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking

[Wikidata-bugs] [Maniphest] T349147: Follow up on rdf-streaming-updater failure 2023-10-17

bking created this task. bking added projects: Wikidata-Query-Service, Data-Platform-SRE. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION We had a brief outage of the RDF streaming updater today. Users could have been impacted from about ~2022 - 2050 UTC. Specifically

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a comment. We could also deploy via a new namespace, but I wonder what implications that would have for our monitoring/tooling etc. Open to feedback/suggestions on this one. TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a comment. Create a savepoint by incrementing the nonce value in the helmfile.d/dse-k8s-services/values.yaml and deploy Destroy the deployment on the dse-k8s cluster /srv/deployment-charts/helmfile.d/dse-k8s-services/rdf-streaming-updater/$ helmfile -e dse-k8s

[Wikidata-bugs] [Maniphest] T337013: [Epic] Splitting the graph in WDQS

bking closed subtask T347505: Prepare new WDQS hosts for graph splitting as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T337013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: dr0ptp4kt, RKemper, bking, tfmorris, elal

[Wikidata-bugs] [Maniphest] T347505: Prepare new WDQS hosts for graph splitting

bking closed this task as "Resolved". bking moved this task from In Progress to Done on the Data-Platform-SRE board. bking added a comment. Work is complete...resolving. TASK DETAIL https://phabricator.wikimedia.org/T347505 WORKBOARD https://phabricator.wikimedia.org/project/

[Wikidata-bugs] [Maniphest] T349147: Follow up on rdf-streaming-updater failure 2023-10-17

bking added a subscriber: dcausse. bking added a comment. Deployed `flink-1.16.1-rdf-0.3.136` release for WCQS and WDQS staging, via: python3 flink/flink-job.py \ --env staging \ --job-name "WDQS Streaming Updater" \ deploy \

[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model

bking added a subtask: T344614: Add Zookeeper config to 'flink-app' test service. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Gehel, BTullis, JMeybohm, gmodena, Ottomata, bking

  1   2   3   4   >