[Wikidata-bugs] [Maniphest] [Commented On] T241291: Simplify WDQS Packaging

2020-01-21 Thread Mstyles
Mstyles added a comment. @addshore that's correct, after removing the gui submodule, I won't be doing any further work TASK DETAIL https://phabricator.wikimedia.org/T241291 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: WMDE-leszek

[Wikidata-bugs] [Maniphest] [Commented On] T241291: Simplify WDQS Packaging

2020-02-10 Thread Mstyles
Mstyles added a comment. @Jdforrester-WMF WMDE will be taking on responsibility for any new deployment methods. That work will be tracked in T192006 <https://phabricator.wikimedia.org/T192006> and T210286 <https://phabricator.wikimedia.org/T210286>. TASK DE

[Wikidata-bugs] [Maniphest] [Commented On] T232006: LDF service does not Vary responses by Accept, sending incorrect cached responses to clients

2020-01-10 Thread Mstyles
Mstyles added a comment. @gehel I think we can consider this closed unless someone is able to reproduce TASK DETAIL https://phabricator.wikimedia.org/T232006 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Mstyles, Gehel, BBlack

[Wikidata-bugs] [Maniphest] [Commented On] T237165: LDF server has 404 errors for JS and CSS resources

2020-01-09 Thread Mstyles
Mstyles added a comment. for clarification the correct response will contain a list that looks like this @prefix schema: <http://schema.org/> . @prefix pq:<http://www.wikidata.org/prop/qualifier/> . @prefix pr:<http://www.wikidata.org/prop/reference/>

[Wikidata-bugs] [Maniphest] [Updated] T241291: Simplify WDQS Packaging

2020-01-16 Thread Mstyles
Mstyles added a comment. After a bunch of discussion with the team, it's been decided that removing the gui submodule from the RDF repository will suffice for now. That will fix our broken build issues (see https://phabricator.wikimedia.org/T242640) @Ladsgroup I definitely think you should

[Wikidata-bugs] [Maniphest] [Updated] T237165: LDF server has 404 errors for JS and CSS resources

2020-01-08 Thread Mstyles
Mstyles added a project: Traffic. Restricted Application added a project: Operations. TASK DETAIL https://phabricator.wikimedia.org/T237165 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Vahurzpu, Lucas_Werkmeister_WMDE, WikidataFacts

[Wikidata-bugs] [Maniphest] [Commented On] T237165: LDF server has 404 errors for JS and CSS resources

2020-01-08 Thread Mstyles
Mstyles added a comment. from inside any of the WDQS machines ( 'wdqs1004.eqiad.wmnet','wdqs1005.eqiad.wmnet', 'wdqs1006.eqiad.wmnet','wdqs1007.eqiad.wmnet') the following curls return the correct data `curl localhost:80/bigdata/ldf` -> direct to nginx server on host `curl localh

[Wikidata-bugs] [Maniphest] [Claimed] T247123: Migrate wikidata-query-rdf-release-silent release job to Docker

2020-04-09 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T247123 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Addshore, hashar, Aklapper, Jdforrester-WMF, Oblanco79, Alter-paule, Beast1978, CBogen, Un1tY, Hook696

[Wikidata-bugs] [Maniphest] [Updated] T247123: Migrate wikidata-query-rdf-release-silent release job to Docker

2020-04-20 Thread Mstyles
Mstyles edited projects, added Discovery-Search (Current work); removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T247123 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Addshore, hashar, Aklapper, Jdforrester-WMF

[Wikidata-bugs] [Maniphest] [Reopened] T247123: Migrate wikidata-query-rdf-release-silent release job to Docker

2020-04-29 Thread Mstyles
Mstyles reopened this task as "Open". TASK DETAIL https://phabricator.wikimedia.org/T247123 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Addshore, hashar, Aklapper, Jdforrester-WMF, Blissjay007, Oblanco79, Alter-paule,

[Wikidata-bugs] [Maniphest] [Commented On] T247123: Migrate wikidata-query-rdf-release-silent release job to Docker

2020-04-29 Thread Mstyles
Mstyles added a comment. @hashar the jenkins job failed due to no git auth to push. you can see it here: https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker-wdqs/3/ also is there a way to only have this job run when manually triggered and not after every merge

[Wikidata-bugs] [Maniphest] [Commented On] T247123: Migrate wikidata-query-rdf-release-silent release job to Docker

2020-05-04 Thread Mstyles
Mstyles added a comment. I know what that is, I forgot to add the archiva deployment profile. I'll put a patch out TASK DETAIL https://phabricator.wikimedia.org/T247123 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Addshore, hashar

[Wikidata-bugs] [Maniphest] [Closed] T247123: Migrate wikidata-query-rdf-release-silent release job to Docker

2020-05-04 Thread Mstyles
Mstyles closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T247123 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Gehel, Ottomata, Addshore, hashar, Aklapper, Jdforrester-WMF, Blissjay007, Oblanc

[Wikidata-bugs] [Maniphest] [Commented On] T247123: Migrate wikidata-query-rdf-release-silent release job to Docker

2020-05-04 Thread Mstyles
Mstyles added a comment. Docs are updated -> https://wikitech.wikimedia.org/wiki/Wikidata_query_service#Production_Deployment and verified that the zip file from the latest version can be downloaded from Archiva TASK DETAIL https://phabricator.wikimedia.org/T247123 EMAIL PREFEREN

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T247123: Migrate wikidata-query-rdf-release-silent release job to Docker

2020-05-04 Thread Mstyles
Mstyles added a subscriber: Gehel. Mstyles added a comment. Talked to @Gehel and the issue is probably that the archiva credentials that come from analytics have different server ID's than the ones we use. https://github.com/wikimedia/wikimedia-discovery-discovery-parent-pom/blob/master

[Wikidata-bugs] [Maniphest] [Claimed] T251515: Automate data reload for SPARQL Endpoint for Commons

2020-05-18 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T251515 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Lea_Lacroix_WMDE, Gehel, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T251515: Automate data reload for SPARQL Endpoint for Commons

2020-05-18 Thread Mstyles
Mstyles added a subscriber: Zbyszko. Mstyles added a comment. As discussed in email with @Zbyszko the script should do the following 1. Download the newest dump (maybe - we can provide it manually, but where's the fun in that?) 2. Munge the data (running a script with appropriate data

[Wikidata-bugs] [Maniphest] T256949: The streaming updater should support suppressed deletes

2020-10-08 Thread Mstyles
Mstyles added a comment. I don't see a schema for suppressed deletes in the list of page schemas (https://schema.wikimedia.org/repositories/primary/jsonschema/mediawiki/page/). Is that information intentionally private? Also, I can't see how having the page namespace and other information

[Wikidata-bugs] [Maniphest] T264006: Deploy Flink to kubernetes (k8s)

2020-10-08 Thread Mstyles
Mstyles added a comment. We're having an in person meeting soon, but I just wanted to outline some things I think will need to happen from a high level. This is based on reading https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Components. 1. Create blubberfiles for both flink

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-10-13 Thread Mstyles
Mstyles moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. Mstyles closed this task as "Resolved". Mstyles added a comment. All patches are merged and items can be deleted successfully TASK DETAIL https://phabricator.wikimedia.o

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-13 Thread Mstyles
Mstyles closed subtask T256875: The streaming updater should support page deletions as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Thadguidry, tfmorris, revi, Ladsgroup

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-10-13 Thread Mstyles
Mstyles reopened this task as "Open". TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, dcausse, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, dar

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-13 Thread Mstyles
Mstyles reopened subtask T256875: The streaming updater should support page deletions as Open. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Thadguidry, tfmorris, revi, Ladsgroup

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

2020-10-19 Thread Mstyles
Mstyles moved this task from All WDQS-related tasks to Current work on the Wikidata-Query-Service board. Mstyles added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T265512 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL

[Wikidata-bugs] [Maniphest] T265526: Create Helm Chart

2020-10-19 Thread Mstyles
Mstyles moved this task from All WDQS-related tasks to Current work on the Wikidata-Query-Service board. Mstyles added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T265526 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-10-06 Thread Mstyles
Mstyles added a comment. Delete functionality is complete, optimization work in progress to ensure that patches created on the streaming updater consumer that get sent to blazegraph do not contain statements from entities that will be deleted TASK DETAIL https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

2020-10-14 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service. TASK DESCRIPTION We need to create a blubberfile <https://wikitech.wikimedia.org/wiki/Blubber/User_Guide#Blubberfiles> in order to use the WMF pipeline to deploy to Kubernetes. Blubberfiles are higher

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

2020-10-14 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service. TASK DESCRIPTION We need to define the pipeline configuration <https://wikitech.wikimedia.org/wiki/PipelineLib/Guides/How_to_configure_CI_for_your_project> for WDQS. This is done by creating a confi

[Wikidata-bugs] [Maniphest] T265526: Create Helm Chart

2020-10-14 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service. TASK DESCRIPTION Helm Charts are needed to configure the Kubernetes Deployment. This include a Helmfile which is needed for deployment to prodcution. Helm charts are stored in this gerrit repo <ht

[Wikidata-bugs] [Maniphest] T256949: The streaming updater should support suppressed deletes

2020-10-14 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T256949 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Mstyles, Bugreporter, dcausse, Aklapper, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

2020-10-14 Thread Mstyles
Mstyles added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, darthmon_wmde, Nandana

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

2020-10-14 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

2020-10-19 Thread Mstyles
Mstyles added a comment. In order to test the image created by the blubberfile, do the following in the rdf repo `cd .pipeline` `blubber blubber.yaml production | docker build --tag blubber-flink-test- --file - .` -> this builds the image and tags it `docker run blubber-flink-t

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

2020-10-19 Thread Mstyles
Mstyles added a comment. The original Flink dockerfile <https://github.com/apache/flink-docker/blob/149ebb09d9146ec46c6df0ab73d638c4d7549c10/1.11/scala_2.11-java8-debian/Dockerfile> exposes ports, which is not an option in blubber. I'm hoping that any networking/ports issues can be re

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-08-17 Thread Mstyles
Mstyles added a comment. Clearing the state is out of the question, I think. Right now, we mark the revision we see a delete at and then we don't allow a Rev Create Event. However, this means that any regular RevCreate events don't allow the item to be reimported ever again. I think

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-08-26 Thread Mstyles
Mstyles added a comment. Wrapping up the flink pipeline work, a patch is out for the integration tests and there's some more test cases to be added for clarity on the Decide Mutation Operation to ensure that all delete cases are covered. Pipeline consumer work is up next TASK DETAIL

[Wikidata-bugs] [Maniphest] T262907: EXISTS inside BIND always false when hint included

2020-09-21 Thread Mstyles
Mstyles triaged this task as "Low" priority. TASK DETAIL https://phabricator.wikimedia.org/T262907 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Svizel_pritula, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlo

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-09-28 Thread Mstyles
Mstyles added a comment. Wrapping up the last phase of deleting an entity from blazegraph. I have verified that items to be deleted are present in the streaming updater output. TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Commented On] T251500: oAuth authentication for SPARQL Endpoint for Commons

2020-05-28 Thread Mstyles
Mstyles added a comment. Here is the nginx example -> https://github.com/nginxinc/NGINX-Demos/tree/master/oauth2-token-introspection-oss TASK DETAIL https://phabricator.wikimedia.org/T251500 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferen

[Wikidata-bugs] [Maniphest] [Commented On] T251500: oAuth authentication for SPARQL Endpoint for Commons

2020-05-22 Thread Mstyles
Mstyles added a comment. Some notes for when/if we do oauth Docs -> https://www.mediawiki.org/wiki/OAuth/For_Developers we might be able to use this npm package <https://www.npmjs.com/package/passport-mediawiki-oauth> for auth with mediawiki This <https://github.com/SQL-en

[Wikidata-bugs] [Maniphest] T265526: Create Helm Chart

2020-10-26 Thread Mstyles
Mstyles updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T265526 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86

[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink

2020-10-26 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search (Current work). TASK DESCRIPTION Currently we are downloading Flink from Apache via wget. Sometimes the gpg key check fails, and the blubber build has to be restarted. Additionally, if we

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-08-03 Thread Mstyles
Mstyles added a comment. Shared model delete functionality is complete, currently in the process of adding delete functionality to the flink pipeline TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-08-05 Thread Mstyles
Mstyles added a comment. What do we want to do if we get a PageDelete event for the same item multiple times? Right now we're not tracking deletes, so I'm assuming multiple deletes for the same item probably can't do that much harm. TASK DETAIL https://phabricator.wikimedia.org/T256875

[Wikidata-bugs] [Maniphest] T257314: Add a Logout link to Wikimedia Commons Query Service

2020-07-14 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T257314 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: CBogen, Lucas_Werkmeister_WMDE, Aklapper, Zbyszko, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi

[Wikidata-bugs] [Maniphest] [Commented On] T251514: UI for SPARQL Endpoint for Commons

2020-06-30 Thread Mstyles
Mstyles added a comment. Since the gui directory is overwritten by scap via symlinks, Guillaume proposed that the config files live in /etc/config and that they be symlinked to the gui directory. See more discussion here:https://gerrit.wikimedia.org/r/c/operations/puppet/+/606297. I want

[Wikidata-bugs] [Maniphest] [Updated] T251514: UI for SPARQL Endpoint for Commons

2020-06-08 Thread Mstyles
Mstyles added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T251514 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: CBogen, Lucas_Werkmeister_WMDE, Zbyszko, WMDE-leszek, Addshore, Aklapper

[Wikidata-bugs] [Maniphest] T251515: Automate data reload for SPARQL Endpoint for Commons

2020-07-29 Thread Mstyles
Mstyles removed Mstyles as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T251515 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Mstyles, dcausse, Zbyszko, Aklapper, Lea_Lacroix_WMDE, Gehel, CBogen, Akuckartz

[Wikidata-bugs] [Maniphest] T251488: [epic] Create minimal SPARQL Endpoint for Commons on WMCS

2020-07-27 Thread Mstyles
Mstyles closed subtask T251515: Automate data reload for SPARQL Endpoint for Commons as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T251488 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Nintendofan885, Zbyszko, Smalyshev

[Wikidata-bugs] [Maniphest] T251515: Automate data reload for SPARQL Endpoint for Commons

2020-07-27 Thread Mstyles
Mstyles closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T251515 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Zbyszko, Aklapper, Lea_Lacroix_WMDE, Gehel, CBogen, Akuckartz, darthmon_wmd

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-07-27 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, dcausse, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

2020-07-27 Thread Mstyles
Mstyles added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, dcausse, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

2020-11-20 Thread Mstyles
Mstyles moved this task from Ready for Development to Needs review on the Discovery-Search (Current work) board. Mstyles added a subscriber: akosiaris. Mstyles added a comment. @akosiaris it was unclear to me whether we need the promote section in the pipeline config. I'm referring

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

2020-12-07 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search (Current work). TASK DESCRIPTION As a production service, the flink streaming updater job needs somewhere to send events that are late/failed/spurious. Currently these go to HDFS but apps

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

2020-12-07 Thread Mstyles
Mstyles added a subscriber: Ottomata. Mstyles added a comment. @Ottomata also suggested via IRC to consider using the event platform instead of kafka TASK DETAIL https://phabricator.wikimedia.org/T269619 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

2020-12-08 Thread Mstyles
Mstyles moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. Mstyles added a comment. I definitely agree that pipeline lib could be used for Java projects, but adding that current functionality with this project is out of scope. We are moving

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

2020-11-23 Thread Mstyles
Mstyles added a comment. Instead of trying to skip CI for every commit, the easiest thing to do would be to move the pipeline directory into its own repo. It's not using any of the code in the current repo anyways. TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

2020-11-23 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: dduvall, jeena, akosiaris, Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

2020-11-24 Thread Mstyles
Mstyles added a comment. The projects are all related and there are dependencies. Separating the streaming-updater-producer would actually be a significant task. I think that would be a separate project to possibly consider in the future. TASK DETAIL https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

2020-11-23 Thread Mstyles
Mstyles added a comment. For #1, that's is correct. We are currently downloading Flink from the internet. I don't think that's the best idea long term, but it seems fine for now. Currently for #2, "Build and verify streaming-updater-producer from source, tracking changes to

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

2020-12-02 Thread Mstyles
Mstyles moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. Mstyles added a comment. blubberfile is done and the docker image is present in the wikimedia docker repository TASK DETAIL https://phabricator.wikimedia.org/T265504 WORKBOARD

[Wikidata-bugs] [Maniphest] T265526: Create Helm Chart

2020-12-10 Thread Mstyles
Mstyles updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T265526 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301

[Wikidata-bugs] [Maniphest] T269876: Add swift plugin to Flink k8s

2020-12-10 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search (Current work). TASK DESCRIPTION Flink needs somewhere to store checkpoint data. We've decided to go with Swift as discussed here: https://phabricator.wikimedia.org/T246004. Now

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

2020-11-09 Thread Mstyles
Mstyles added a comment. @akosiaris I started using the new Java images that you uploaded. I wasn't able to install gpg in the build process. There are some conflicts. We can skip gpg verification of the Flink tar, but I don't think that's a good idea. I will continue to do some debugging

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

2020-11-02 Thread Mstyles
Mstyles added a comment. @akosiaris when you get some time, can you please take another look at https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/635074 TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T265526: Create Helm Chart

2020-11-02 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T265526 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE

[Wikidata-bugs] [Maniphest] T267825: Outdated data still present in WCQS a month after statement update

2021-01-04 Thread Mstyles
Mstyles added a comment. On the wcqs beta host, `curl -d "query=select * { sdc:M8979671 wdt:P571 ?o . }" localhost/bigdata/namespace/wcq/sparql` returns an item with the correct data and `curl -d "query=select * { sdc:M8979671 wdt:P571 ?o . }" localhost:/bigdata/

[Wikidata-bugs] [Maniphest] T270371: wikimedia-event-utilities should provide tools for JVM based apps producing directly to kafka

2021-01-04 Thread Mstyles
Mstyles assigned this task to dcausse. TASK DETAIL https://phabricator.wikimedia.org/T270371 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Mstyles Cc: Ottomata, dcausse, Aklapper, MPhamWMF, CBogen, Akuckartz, 4748kitoko, holger.knust

[Wikidata-bugs] [Maniphest] T272447: Extract a list of the 200 most viewed black historical figures from WDQS

2021-01-20 Thread Mstyles
Mstyles added a comment. @dcausse and I discussed getting the page view counts from https://pageviews.toolforge.org/?project=en.wikipedia.org=all-access=user=0=latest-20=Cat|Dog unless you have another suggestion/idea @Miriam TASK DETAIL https://phabricator.wikimedia.org/T272447 EMAIL

[Wikidata-bugs] [Maniphest] T272447: Extract a list of the 200 most viewed black historical figures from WDQS

2021-01-20 Thread Mstyles
Mstyles added a comment. also @Miriam both queries still return wikipedia pages that do have images TASK DETAIL https://phabricator.wikimedia.org/T272447 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: MPhamWMF, CBogen, Miriam

[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink

2021-01-25 Thread Mstyles
Mstyles added a comment. In the interim (until we have the deb package) @dcausse suggested putting the Flink packages on people.wikimedia.org since packages on the official Flink download sites get updated and removed pretty frequently. TASK DETAIL https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] T267825: Outdated data still present in WCQS a month after statement update

2021-01-04 Thread Mstyles
Mstyles added a comment. The dumps have been investigated and the data is correct there, the problem is definitely on the search team's side. It could be with the reload scripts or nginx. It was possible to see the incorrect data on the wcqs server, so the problem is most likely with nginx

[Wikidata-bugs] [Maniphest] T273097: Create Flink Base Image

2021-01-29 Thread Mstyles
Mstyles added a subscriber: dcausse. Mstyles added a comment. The problem is that we need more than just the jars. We need the entire tar file. @dcausse suggested people.wikimedia.org but that was considered not a good place for CI to hit TASK DETAIL https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] T273097: Create Flink Base Image

2021-01-28 Thread Mstyles
Mstyles renamed this task from "Create Flink BaseImage" to "Create Flink Base Image". TASK DETAIL https://phabricator.wikimedia.org/T273097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, akosiari

[Wikidata-bugs] [Maniphest] T273097: Create Flink BaseImage

2021-01-28 Thread Mstyles
Mstyles renamed this task from "Create Base Flink Image" to "Create Flink BaseImage". TASK DETAIL https://phabricator.wikimedia.org/T273097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, akosiari

[Wikidata-bugs] [Maniphest] T273095: Deploy Helm Chart

2021-01-27 Thread Mstyles
Mstyles updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T273095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi

[Wikidata-bugs] [Maniphest] T273095: Deploy Helm Chart

2021-01-27 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search (Current work), Patch-For-Review. Restricted Application removed a project: Patch-For-Review. TASK DESCRIPTION In order for the helm chart to be deployed, the helmfile.d structure needs

[Wikidata-bugs] [Maniphest] T273095: Deploy Helm Chart

2021-01-27 Thread Mstyles
Mstyles updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T273095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi

[Wikidata-bugs] [Maniphest] T273098: High Availability Flink

2021-01-27 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search (Current work), Patch-For-Review. Restricted Application removed a project: Patch-For-Review. TASK DESCRIPTION With the latest updated of Flink (1.12), High Availability is now supported

[Wikidata-bugs] [Maniphest] T273097: Create Base Flink Image

2021-01-27 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search (Current work), Patch-For-Review. Restricted Application removed a project: Patch-For-Review. TASK DESCRIPTION This is an alternative to https://phabricator.wikimedia.org/T266495. Flink

[Wikidata-bugs] [Maniphest] T269876: Add swift plugin to Flink k8s

2021-01-27 Thread Mstyles
Mstyles added a comment. The swift plugin has been added to the docker image and is properly configured in the helm chart. TASK DETAIL https://phabricator.wikimedia.org/T269876 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper

[Wikidata-bugs] [Maniphest] T265526: Create Helm Chart

2021-01-27 Thread Mstyles
Mstyles updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T265526 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Mirahamira, Aklapper, Gehel, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi

[Wikidata-bugs] [Maniphest] T265526: Create Helm Chart

2021-01-27 Thread Mstyles
Mstyles added a comment. Updating this task to remove the helmfile requirement. That is more of a final step that doesn't include the helm chart. The benchmarking that's been highlighted is not possible on a local machine, we will need to do proper load testing in a staging environment

[Wikidata-bugs] [Maniphest] T273097: Create Flink Base Image

2021-02-08 Thread Mstyles
Mstyles moved this task from Ready for Development to Needs Reporting on the Discovery-Search (Current work) board. Mstyles added a comment. Not being done since no longer needed TASK DETAIL https://phabricator.wikimedia.org/T273097 WORKBOARD https://phabricator.wikimedia.org/project

[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink

2021-02-08 Thread Mstyles
Mstyles closed this task as "Declined". TASK DETAIL https://phabricator.wikimedia.org/T266495 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: dcausse, elukey, EBernhardson, Aklapper, Gehel, Mstyles, Ottomata, MPhamWMF, CBogen,

[Wikidata-bugs] [Maniphest] T264006: Deploy Flink to kubernetes (k8s)

2021-02-08 Thread Mstyles
Mstyles closed subtask T266495: Create Debian Package for Flink as Declined. TASK DETAIL https://phabricator.wikimedia.org/T264006 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: akosiaris, Mstyles, Gehel, Aklapper, MPhamWMF, Alter-paule

[Wikidata-bugs] [Maniphest] T273097: Create Flink Base Image

2021-02-04 Thread Mstyles
Mstyles added a comment. @akosiaris I just learned that there are archive links that have all of the Flink packages. I'm proposing that we close both this ticket and https://phabricator.wikimedia.org/T266495 and just use the Flink archive links where we won't have to worry about

[Wikidata-bugs] [Maniphest] T272265: Resolve WDQS changelog situation

2021-02-09 Thread Mstyles
Mstyles claimed this task. Mstyles moved this task from Ready for Development to In Progress on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T272265 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink

2021-02-01 Thread Mstyles
Mstyles updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T266495 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: dcausse, elukey, EBernhardson, Aklapper, Gehel, Mstyles, Ottomata, MPhamWMF, CBogen, Akuckartz

[Wikidata-bugs] [Maniphest] T273098: High Availability Flink

2021-03-24 Thread Mstyles
Mstyles edited subscribers, added: jijiki, JMeybohm; removed: Mstyles. Mstyles added a comment. The changes for HA are mostly changing a few lines of the config file. The part that is unclear is whether we can get a service account for pods that allow them to do CRUD operations

[Wikidata-bugs] [Maniphest] T278385: Streaming Updater must make all requests to proxy endpoints

2021-03-24 Thread Mstyles
Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search (Current work), Patch-For-Review. Restricted Application removed a project: Patch-For-Review. TASK DESCRIPTION All applications deployed in the Kubernetes cluster must make their requests

[Wikidata-bugs] [Maniphest] T278385: Streaming Updater must make all requests to proxy endpoints

2021-03-30 Thread Mstyles
Mstyles added a comment. I wanted to clarify for Swift as well, will there be proxies to connect to the swift auth url and connecting the the thanos swift cluster? TASK DETAIL https://phabricator.wikimedia.org/T278385 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T278385: Streaming Updater must make all requests to proxy endpoints

2021-04-06 Thread Mstyles
Mstyles added a comment. Thanks! TASK DETAIL https://phabricator.wikimedia.org/T278385 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: dcausse, JMeybohm, jijiki, Aklapper, Gehel, akosiaris, Mstyles, Invadibot, MPhamWMF, maantietaja

[Wikidata-bugs] [Maniphest] T278385: Streaming Updater must make all requests to proxy endpoints

2021-03-31 Thread Mstyles
Mstyles added a subscriber: dcausse. Mstyles added a comment. @akosiaris the swift proxies in there seem are pointing to the swift cluster that we use (thano-swift). We'll need additional proxies set up for that as well. Would it be more helpful to have a separate ticket for the proxies

[Wikidata-bugs] [Maniphest] T264006: Deploy Flink to kubernetes (k8s)

2021-03-12 Thread Mstyles
Mstyles added a subtask: Restricted Task. TASK DETAIL https://phabricator.wikimedia.org/T264006 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: akosiaris, Mstyles, Gehel, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen

[Wikidata-bugs] [Maniphest] T273095: Deploy Helm Chart

2021-03-11 Thread Mstyles
Mstyles added a comment. currently blocked by a helm test issue described in https://phabricator.wikimedia.org/T277252 TASK DETAIL https://phabricator.wikimedia.org/T273095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper

[Wikidata-bugs] [Maniphest] T264006: Deploy Flink to kubernetes (k8s)

2021-03-18 Thread Mstyles
Mstyles updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T264006 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: akosiaris, Mstyles, Gehel, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY

[Wikidata-bugs] [Maniphest] T264006: Deploy Flink to kubernetes (k8s)

2021-03-18 Thread Mstyles
Mstyles updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T264006 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: akosiaris, Mstyles, Gehel, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY

[Wikidata-bugs] [Maniphest] T273098: High Availability Flink

2021-03-18 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T273098 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi

[Wikidata-bugs] [Maniphest] T273095: Deploy Helm Chart

2021-02-25 Thread Mstyles
Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T273095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86

  1   2   >