Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T239908
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, dcausse, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86,
Lucas_Werkmeister_WMDE, GoranSMilovanovic
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
[ ] Verify test coverage for Updater
[ ] Create E2E test
[ ] Review previous tests ( remove
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T230588
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Larske, Mathew.onipe, Esc3300, agray, Jakub.klimek, Gehel, bennofs, Magnus,
Robby, Mmarx, jeremyb
Zbyszko added a comment.
@Lucas_Werkmeister_WMDE deployment done, can you verify the fix on
production?
TASK DETAIL
https://phabricator.wikimedia.org/T245637
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Zbyszko, dcausse
Zbyszko closed this task as "Declined".
Zbyszko added a comment.
We're going in the new direction of rewriting updated into a streaming
applicaiton.
TASK DETAIL
https://phabricator.wikimedia.org/T231411
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailp
Zbyszko closed subtask T231411: Test new Updater service as
Declined.
TASK DETAIL
https://phabricator.wikimedia.org/T212826
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Igorkim78, Zbyszko
Cc: Gehel, Abbe98, EgonWillighagen, Fnielsen
Zbyszko created this task.
Zbyszko added projects: Wikidata-Query-Service, Wikidata.
TASK DESCRIPTION
We need a Flink task to communicate with Wikibase API, to get entity data
based on revision. Task should be pluggable into pipeline as an Async I/O task.
TASK DETAIL
https
Zbyszko closed subtask T245727: Create a streaming-updater submodule under
query/wikidata/rdf as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T244590
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Iamamz3, Smalyshev, Ottomata
Zbyszko closed this task as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T245727
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Gehel, Zbyszko, Aklapper, JAllemandou, Ottomata, Smalyshev, Iamamz3,
dcausse, dar
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T247058
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, dcausse, Zbyszko, Gehel, darthmon_wmde, Legado_Shulgin, Nandana,
Davinaclare77, Qtn1293
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T247058
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, dcausse, Zbyszko, Gehel, darthmon_wmde, Legado_Shulgin, Nandana,
Davinaclare77, Qtn1293
Zbyszko created this task.
Zbyszko added projects: Release-Engineering-Team, Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
We want to be able to do releases of wikidata/query/rdf. To do that we
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
We want to have a similar way of deploying WDQS to Archiva as Analytics do
here:
https
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T105427
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Bugreporter, Sjoerddebruin, Krenair, gerritbot, JanZerebecki,
Deskana, daniel, Legoktm, Aklapper
Zbyszko changed the task status from "Open" to "Stalled".
Zbyszko added a comment.
We are stopping work on this for now, pending architecture change. We have a
solution in mind that can make this work automatically - based on the events
coming from mediawiki.revision-visi
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T241213
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86,
Lucas_Werkmeister_WMDE
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T249099
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86,
Lucas_Werkmeister_WMDE, GoranSMilovanovic
Zbyszko added a comment.
After adding some additional logging, this is what we see:
> 2020-04-08 07:44:07,287 ERROR
org.wikidata.query.rdf.updater.ExtractTriplesFunction - Exception
thrown for op: FullImport(Property:P1331
<https://phabricator.wikimedia.org/P1331>,2020
Zbyszko added a comment.
Will be done for the streaming updater.
TASK DETAIL
https://phabricator.wikimedia.org/T241213
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86
Zbyszko closed this task as "Declined".
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T241213
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, Zbyszko, darthmon_wmde, Nan
Zbyszko renamed this task from "Implement NTriples format ouput in Streaming
Updater" to "Implement ouput format in Streaming Updater".
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T248464
EMAIL PREFERENCES
https://phabricator.wi
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T248449
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86,
Lucas_Werkmeister_WMDE, GoranSMilovanovic
Zbyszko renamed this task from "Monitor Streaming Updater Latency" to "Monitor
Streaming Updater - metrics ".
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T248450
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/pa
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T248451
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, darthmon_wmde, Nandana, Lahi, Gq86,
Lucas_Werkmeister_WMDE, GoranSMilovanovic
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
Early tests showed that checkpointing doesn't work very well, during 4 days
run, statistics
Zbyszko renamed this task from "Monitor Streaming Updater - metrics " to "[WDQS
Streaming Updater] Monitor Streaming Updater - metrics ".
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T248450
EMAIL PREFERENCES
https://phabricator.wikimedi
Zbyszko renamed this task from "Custom parallelism configuration for Streaming
Updater" to "[WDQS Streaming Updater] Custom parallelism configuration for
Streaming Updater".
TASK DETAIL
https://phabricator.wikimedia.org/T248451
EMAIL PREFERENCES
https://phabricator.wi
Zbyszko renamed this task from "Implement ouput format in Streaming Updater" to
"[WDQS Streaming Updater] Implement ouput format in Streaming Updater".
TASK DETAIL
https://phabricator.wikimedia.org/T248464
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/pa
Zbyszko renamed this task from "Deploy and configure Streaming Updater Hadoop
YARN" to "[WDQS Streaming Updater] Deploy and configure Streaming Updater
Hadoop YARN".
TASK DETAIL
https://phabricator.wikimedia.org/T248452
EMAIL PREFERENCES
https://phabricator.wikimedi
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
Pipeline breaks with a following exception
Zbyszko renamed this task from "Add error handling for Streaming Updater" to
"[WDQS Streaming Updater] Add error handling for Streaming Updater".
TASK DETAIL
https://phabricator.wikimedia.org/T248449
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/pa
Zbyszko added a comment.
No, right now exceptions are just uncaught.
As for an error itself - triple list doesn't have to be empty. During
munging, statement list is being modified and by the time that this exception
is thrown, it can be empty. I'm not yet suggesting
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
AC: TODO
TASK DETAIL
https://phabricator.wikimedia.org/T248451
EMAIL PREFERENCES
https
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a project: Wikidata.
TASK DETAIL
https://phabricator.wikimedia.org/T248450
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, darthmon_wmde
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
AC: TODO
TASK DETAIL
https://phabricator.wikimedia.org/T248452
EMAIL PREFERENCES
https
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
AC: TODO
TASK DETAIL
https://phabricator.wikimedia.org/T248449
EMAIL PREFERENCES
https
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
AC: TODO
TASK DETAIL
https://phabricator.wikimedia.org/T248464
EMAIL PREFERENCES
https
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
AC:
- Streaming Updater uses WikibaseRepositry
- WikibaseRepository metrics are reported
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
Right now, every functionality related to updating is either in tools/common
(old updater + shared
Zbyszko added a subtask: T251096: [WDQS Streaming Updater] Organise module
structure.
TASK DETAIL
https://phabricator.wikimedia.org/T244590
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: revi, Mholloway, Ladsgroup, Multichill
Zbyszko added a parent task: T244590: EPIC: Rework the WDQS updater as an event
driven application.
TASK DETAIL
https://phabricator.wikimedia.org/T251096
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, darthmon_wmde
Zbyszko renamed this task from "Update blazegraph based on the content present
in the streaming updater output kafka stream" to "[WDQS Streaming Updater]
Update blazegraph based on the content present in the streaming updater output
kafka stream".
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
Currently, the deployer, after deploying WDQS to canary server (wdqs1003) has
to manually perform
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
During last WDQS outage it was difficult to correctly ascertain the impact of
it. Current graphs
Zbyszko added a project: Sustainability (Incident Prevention).
TASK DETAIL
https://phabricator.wikimedia.org/T252503
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86
Zbyszko closed this task as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T249500
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86,
Lucas_Werkme
Zbyszko closed subtask T249500: [WDQS Streaming Updater] Reuse
WikibaseRepository as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T244590
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: revi, Mholloway, Ladsgroup, Multichill
Zbyszko closed this task as "Resolved".
Zbyszko added a comment.
Munger correctly processes commons dump.
TASK DETAIL
https://phabricator.wikimedia.org/T243292
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Mahir256, Ph
Zbyszko closed subtask T243292: Fix the munger to support commons RDF dump as
Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T221917
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: nettrom_WMF, Mahir256, dcausse, EBernhardson
Zbyszko closed subtask T243292: Fix the munger to support commons RDF dump as
Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T251497
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Gehel, CBogen, darthmon_wmde, Nandana
Zbyszko closed subtask T243292: Fix the munger to support commons RDF dump as
Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T243270
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, CBogen, darthmon_wmde
Zbyszko closed this task as "Resolved".
Zbyszko added a comment.
Async I/O has limitations that would complicate the pipeline and in the end
it seems it's not actually needed.
TASK DETAIL
https://phabricator.wikimedia.org/T246417
EMAIL PREFERENCES
https://phabricator.wik
Zbyszko closed subtask T246417: Create Async I/O Flink task for wikibase
communication as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T244590
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: revi, Mholloway, Ladsgroup
Zbyszko closed subtask T248450: [WDQS Streaming Updater] Monitor Streaming
Updater - metrics as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T244590
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: revi, Mholloway, Ladsgroup
Zbyszko closed this task as "Resolved".
Zbyszko added a comment.
Metrics are available on grafana.wikimedia.org through Graphite:
https://grafana.wikimedia.org/d/_kZ1VGRGk/wdqs-pipeline?orgId=1=1m
TASK DETAIL
https://phabricator.wikimedia.org/T248450
EMAIL PREFERENC
Zbyszko added a project: Sustainability (Incident Prevention).
TASK DETAIL
https://phabricator.wikimedia.org/T252504
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86
Zbyszko added a project: Sustainability (Incident Prevention).
TASK DETAIL
https://phabricator.wikimedia.org/T252508
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen, darthmon_wmde, Nandana, Lahi, Gq86
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a project: Wikidata.
TASK DETAIL
https://phabricator.wikimedia.org/T252503
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen
Zbyszko closed this task as "Resolved".
Zbyszko added a comment.
We have a build in Jenkins for that now.
TASK DETAIL
https://phabricator.wikimedia.org/T243603
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: WMDE-leszek
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T252503
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Gehel, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlos314, jijiki
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
As a WDQS/WCQS maintainer a want to be able to access jetty startup logs from
a file in /var/log
Zbyszko added a comment.
@RKemper we should be able to retry categories reload after deploying this.
TASK DETAIL
https://phabricator.wikimedia.org/T261097
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, RKemper, Gehel
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T258240
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlos314, Lahi, Gq86
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
As a user of WCQS I want to have a real-time updates to WCQS so that I can
see the changes soon
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T262265
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314,
Lahi, Gq86
Zbyszko added a comment.
If we go by the solution of having an additional pipeline for SDC -
https://phabricator.wikimedia.org/T262020 should be done first.
TASK DETAIL
https://phabricator.wikimedia.org/T262265
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Zbyszko added a comment.
If we go by the solution of having an additional pipeline for SDC -
https://phabricator.wikimedia.org/T262020 should be done first.
TASK DETAIL
https://phabricator.wikimedia.org/T262020
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Zbyszko added a comment.
Just to be own's devil's advocate or to provide alternatives, we can solve
both downtime and real-time updates with the old updater. Additionally, we can
eliminate the downtime by having two blazegraph instances in an active/standby
setup.
TASK DETAIL
https
Zbyszko added a subtask: T262265: Provide real-time updates for WCQS.
TASK DETAIL
https://phabricator.wikimedia.org/T260568
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: CBogen, Gehel, Aklapper, NavinRizwi, Akuckartz, darthmon_wmde
Zbyszko added a parent task: T260568: [EPIC] Productionize WCQS.
TASK DETAIL
https://phabricator.wikimedia.org/T262265
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana
Zbyszko added a comment.
I went with the fix for wikidata/query/rdf scripts - it made most sense to
me, since issues with a single wiki should block updates for others. Once it's
merged, I'll add an entry on that to runbook
TASK DETAIL
https://phabricator.wikimedia.org/T261097
EMAIL
Zbyszko added a comment.
The process we have right now is that we use SDC dumps to reload the data
each week. Dumps are made each Sunday, which means, that all the changes made
between Aug 30th and Sep 6th will only show up in the dump released on Sep
6th. I pushed the update time from
Zbyszko added a comment.
The process we have right now is that we use SDC dumps to reload the data
each week. Dumps are made each Sunday, which means, that all the changes made
between Aug 30th and Sep 6th will only show up in the dump released on Sep
6th. I pushed the update time from
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T262828
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Gehel, Bugreporter, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde,
Nandana, Namenlos314, Lahi, Gq86
Zbyszko triaged this task as "High" priority.
TASK DETAIL
https://phabricator.wikimedia.org/T262828
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Gehel, Bugreporter, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T261119
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, dcausse, Gehel, CBogen, Akuckartz, darthmon_wmde,
Nandana, Namenlos314, Lahi, Gq86
Zbyszko created this task.
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
As a user I want a shortest possible downtime for WCQS in case of data
reload, so that I can
Zbyszko moved this task from All WDQS-related tasks to Current work on the
Wikidata-Query-Service board.
Zbyszko added a project: Discovery-Search (Current work).
TASK DETAIL
https://phabricator.wikimedia.org/T262828
WORKBOARD
https://phabricator.wikimedia.org/project/board/891/
EMAIL
Zbyszko added a project: Wikidata-Query-Service.
Restricted Application added a project: Wikidata.
TASK DETAIL
https://phabricator.wikimedia.org/T262020
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen, Akuckartz
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T261097
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde,
Legado_Shulgin, Nandana
Zbyszko added a comment.
Today's reload happened and if this (https://tinyurl.com/y5vd95rm) query is
correct, there are no duplicates. @Jarekt, can you confirm?
TASK DETAIL
https://phabricator.wikimedia.org/T262178
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Zbyszko removed Zbyszko as the assignee of this task.
TASK DETAIL
https://phabricator.wikimedia.org/T264042
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Zbyszko, CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek,
Vojtech.dostal
Zbyszko added a comment.
The error you get:
Referenced from:
/private/var/folders/2t/2g54bjr10830rv00508_y13wgn/T/flink-io-6ec39247-613e-410a-a83f-712f841ce3a8/rocksdb-lib-396871de50f5fa7595c1071b59c34498/librocksdbjni-osx.jnilib
(which was built for Mac OS X 10.15)
would
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T264042
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek, Vojtech.dostal,
Akuckartz, darthmon_wmde, Nandana
Zbyszko added a comment.
@fgiunchedi Currently, Flink pipeline resides on the Analytics Hadoop
cluster. As for the question whether Flink creates it's containers - I think
not, it did complain when there was no container, so I assume it expects one.
TASK DETAIL
https
Zbyszko removed a project: Patch-For-Review.
TASK DETAIL
https://phabricator.wikimedia.org/T256949
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Bugreporter, dcausse, Aklapper, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlos314
Zbyszko added a comment.
I'm fine with the thanos cluster option - we can proceed with that. @Ottomata
do you know if thanos swift cluster is accessible from hadoop?
TASK DETAIL
https://phabricator.wikimedia.org/T246004
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T264042
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Zbyszko, CBogen, Lucas_Werkmeister_WMDE, Aklapper, matej_suchanek,
Vojtech.dostal, Akuckartz, darthmon_wmde
Zbyszko added a comment.
It's currently not possible - as long as Wikimedia Commons Query Service is
in beta, OAuth authorization will not work with federation. We want to
reevaluate this federation once we productionize WCQS.
TASK DETAIL
https://phabricator.wikimedia.org/T265891
EMAIL
Zbyszko added a comment.
First look at the issue - usual culpruits don't seem to apply here:
- Munged dump is correct for one of the affected entitites
- query, after loading the data into blazegraph is fine,too
Interestingly, every single entity I found with this, was updated
Zbyszko added a comment.
According to this query:
SELECT DISTINCT ?item ?rev ?date WHERE {
{
?st ps:P50|ps:P106|ps:P136|ps:P275|pq:P512|pq:P106
<http://commons.wikimedia.org/wiki/Special:FilePath/>.
?item ?p ?st
}
Zbyszko added a comment.
All the entities affected were refreshed and this:
SELECT ?p (COUNT(*) AS ?count) WHERE {
?s ?p <http://commons.wikimedia.org/wiki/Special:FilePath/>.
}
GROUP BY ?p
ORDER BY DESC(?count)
no longer returns any results.
All af
Zbyszko added a comment.
We lack precise data for production - we haven't really optimised yet and
complete functionality isn't yet ready (it will soon, though). Rarely, we get
around 8-9GB checkpoints (when bootstrapping for example), but they do not
happen regularly. Normally, checkpoints
Zbyszko added a comment.
In T248449#6411542 <https://phabricator.wikimedia.org/T248449#6411542>,
@dcausse wrote:
> In T248449#6382230 <https://phabricator.wikimedia.org/T248449#6382230>,
@Zbyszko wrote:
>
>> We need to decide our approach on possible data c
Zbyszko added a comment.
During the first data reload for some reason there data was not restored
properly. I couldn't find a root cause of this - I'm doing some small changes
to have a better understanding of the issue if it happens again.
TASK DETAIL
https://phabricator.wikimedia.org
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T248449
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlos314, Lahi, Gq86
Zbyszko claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T251096
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314,
Lahi, Gq86
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T248449
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlos314, Lahi, Gq86
Zbyszko updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T248449
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko
Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlos314, Lahi, Gq86
Zbyszko added a comment.
@fgiunchedi We estimate we'd need around 500GB of storage for the streaming
updater (not accounting for replicas). Our use case is almost always write only
(checkpoints are read only on pipeline restarts, which ideally will be done
rarely) - but we have a elasticity
1 - 100 of 306 matches
Mail list logo
