Mstyles added a comment.
In the interim (until we have the deb package) @dcausse suggested putting the
Flink packages on people.wikimedia.org since packages on the official Flink
download sites get updated and removed pretty frequently.
TASK DETAIL
https://phabricator.wikimedia.org
Mstyles added a comment.
also @Miriam both queries still return wikipedia pages that do have images
TASK DETAIL
https://phabricator.wikimedia.org/T272447
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: MPhamWMF, CBogen, Miriam
Mstyles added a comment.
@dcausse and I discussed getting the page view counts from
https://pageviews.toolforge.org/?project=en.wikipedia.org=all-access=user=0=latest-20=Cat|Dog
unless you have another suggestion/idea @Miriam
TASK DETAIL
https://phabricator.wikimedia.org/T272447
EMAIL
Mstyles assigned this task to dcausse.
TASK DETAIL
https://phabricator.wikimedia.org/T270371
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: dcausse, Mstyles
Cc: Ottomata, dcausse, Aklapper, MPhamWMF, CBogen, Akuckartz, 4748kitoko,
holger.knust
Mstyles added a comment.
On the wcqs beta host, `curl -d "query=select * { sdc:M8979671 wdt:P571 ?o
. }" localhost/bigdata/namespace/wcq/sparql` returns an item with the correct
data and `curl -d "query=select * { sdc:M8979671 wdt:P571 ?o . }"
localhost:/bigdata/
Mstyles added a comment.
The dumps have been investigated and the data is correct there, the problem
is definitely on the search team's side. It could be with the reload scripts or
nginx. It was possible to see the incorrect data on the wcqs server, so the
problem is most likely with nginx
Mstyles created this task.
Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search
(Current work).
TASK DESCRIPTION
Flink needs somewhere to store checkpoint data. We've decided to go with
Swift as discussed here: https://phabricator.wikimedia.org/T246004. Now
Mstyles updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T265526
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz,
Hook696, Kent7301
Mstyles moved this task from Needs review to Needs Reporting on the
Discovery-Search (Current work) board.
Mstyles added a comment.
I definitely agree that pipeline lib could be used for Java projects, but
adding that current functionality with this project is out of scope. We are
moving
Mstyles added a subscriber: Ottomata.
Mstyles added a comment.
@Ottomata also suggested via IRC to consider using the event platform instead
of kafka
TASK DETAIL
https://phabricator.wikimedia.org/T269619
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
Mstyles created this task.
Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search
(Current work).
TASK DESCRIPTION
As a production service, the flink streaming updater job needs somewhere to
send events that are late/failed/spurious. Currently these go to HDFS but apps
Mstyles moved this task from Needs review to Needs Reporting on the
Discovery-Search (Current work) board.
Mstyles added a comment.
blubberfile is done and the docker image is present in the wikimedia docker
repository
TASK DETAIL
https://phabricator.wikimedia.org/T265504
WORKBOARD
Mstyles added a comment.
The projects are all related and there are dependencies. Separating the
streaming-updater-producer would actually be a significant task. I think that
would be a separate project to possibly consider in the future.
TASK DETAIL
https://phabricator.wikimedia.org
Mstyles added a comment.
For #1, that's is correct. We are currently downloading Flink from the
internet. I don't think that's the best idea long term, but it seems fine for
now.
Currently for #2, "Build and verify streaming-updater-producer from source,
tracking changes to
Mstyles added a comment.
Instead of trying to skip CI for every commit, the easiest thing to do would
be to move the pipeline directory into its own repo. It's not using any of the
code in the current repo anyways.
TASK DETAIL
https://phabricator.wikimedia.org/T265512
EMAIL PREFERENCES
Mstyles claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T265512
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: dduvall, jeena, akosiaris, Aklapper, Gehel, Mstyles, Alter-paule,
Beast1978, CBogen, Un1tY, Akuckartz
Mstyles moved this task from Ready for Development to Needs review on the
Discovery-Search (Current work) board.
Mstyles added a subscriber: akosiaris.
Mstyles added a comment.
@akosiaris it was unclear to me whether we need the promote section in the
pipeline config. I'm referring
Mstyles added a comment.
@akosiaris I started using the new Java images that you uploaded. I wasn't
able to install gpg in the build process. There are some conflicts. We can skip
gpg verification of the Flink tar, but I don't think that's a good idea. I will
continue to do some debugging
Mstyles added a comment.
@akosiaris when you get some time, can you please take another look at
https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/635074
TASK DETAIL
https://phabricator.wikimedia.org/T265504
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Mstyles claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T265526
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, Nandana, Namenlos314, Lahi,
Gq86, Lucas_Werkmeister_WMDE
Mstyles created this task.
Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search
(Current work).
TASK DESCRIPTION
Currently we are downloading Flink from Apache via wget. Sometimes the gpg
key check fails, and the blubber build has to be restarted. Additionally, if we
Mstyles updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T265526
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, Nandana, Namenlos314, Lahi,
Gq86
Mstyles added a comment.
The original Flink dockerfile
<https://github.com/apache/flink-docker/blob/149ebb09d9146ec46c6df0ab73d638c4d7549c10/1.11/scala_2.11-java8-debian/Dockerfile>
exposes ports, which is not an option in blubber. I'm hoping that any
networking/ports issues can be re
Mstyles added a comment.
In order to test the image created by the blubberfile, do the following in
the rdf repo
`cd .pipeline`
`blubber blubber.yaml production | docker build --tag
blubber-flink-test- --file - .` -> this builds the image and tags it
`docker run blubber-flink-t
Mstyles moved this task from All WDQS-related tasks to Current work on the
Wikidata-Query-Service board.
Mstyles added a project: Discovery-Search (Current work).
TASK DETAIL
https://phabricator.wikimedia.org/T265512
WORKBOARD
https://phabricator.wikimedia.org/project/board/891/
EMAIL
Mstyles moved this task from All WDQS-related tasks to Current work on the
Wikidata-Query-Service board.
Mstyles added a project: Discovery-Search (Current work).
TASK DETAIL
https://phabricator.wikimedia.org/T265526
WORKBOARD
https://phabricator.wikimedia.org/project/board/891/
EMAIL
Mstyles claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T256949
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Mstyles, Bugreporter, dcausse, Aklapper, Alter-paule, Beast1978, CBogen,
Un1tY, Akuckartz, Hook696
Mstyles claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T265504
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlos314, Lahi, Gq86
Mstyles added a project: Discovery-Search (Current work).
TASK DETAIL
https://phabricator.wikimedia.org/T265504
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, darthmon_wmde, Nandana
Mstyles created this task.
Mstyles added projects: Wikidata, Wikidata-Query-Service.
TASK DESCRIPTION
Helm Charts are needed to configure the Kubernetes Deployment. This include a
Helmfile which is needed for deployment to prodcution. Helm charts are stored
in this gerrit repo
<ht
Mstyles created this task.
Mstyles added projects: Wikidata, Wikidata-Query-Service.
TASK DESCRIPTION
We need to define the pipeline configuration
<https://wikitech.wikimedia.org/wiki/PipelineLib/Guides/How_to_configure_CI_for_your_project>
for WDQS. This is done by creating a confi
Mstyles created this task.
Mstyles added projects: Wikidata, Wikidata-Query-Service.
TASK DESCRIPTION
We need to create a blubberfile
<https://wikitech.wikimedia.org/wiki/Blubber/User_Guide#Blubberfiles> in order
to use the WMF pipeline to deploy to Kubernetes. Blubberfiles are higher
Mstyles reopened subtask T256875: The streaming updater should support page
deletions as Open.
TASK DETAIL
https://phabricator.wikimedia.org/T244590
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Thadguidry, tfmorris, revi, Ladsgroup
Mstyles reopened this task as "Open".
TASK DETAIL
https://phabricator.wikimedia.org/T256875
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, dcausse, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz,
Hook696, dar
Mstyles closed subtask T256875: The streaming updater should support page
deletions as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T244590
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Thadguidry, tfmorris, revi, Ladsgroup
Mstyles moved this task from In Progress to Needs Reporting on the
Discovery-Search (Current work) board.
Mstyles closed this task as "Resolved".
Mstyles added a comment.
All patches are merged and items can be deleted successfully
TASK DETAIL
https://phabricator.wikimedia.o
Mstyles added a comment.
We're having an in person meeting soon, but I just wanted to outline some
things I think will need to happen from a high level. This is based on reading
https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Components.
1. Create blubberfiles for both flink
Mstyles added a comment.
I don't see a schema for suppressed deletes in the list of page schemas
(https://schema.wikimedia.org/repositories/primary/jsonschema/mediawiki/page/).
Is that information intentionally private?
Also, I can't see how having the page namespace and other information
Mstyles added a comment.
Delete functionality is complete, optimization work in progress to ensure
that patches created on the streaming updater consumer that get sent to
blazegraph do not contain statements from entities that will be deleted
TASK DETAIL
https://phabricator.wikimedia.org
Mstyles added a comment.
Wrapping up the last phase of deleting an entity from blazegraph. I have
verified that items to be deleted are present in the streaming updater output.
TASK DETAIL
https://phabricator.wikimedia.org/T256875
EMAIL PREFERENCES
https://phabricator.wikimedia.org
Mstyles triaged this task as "Low" priority.
TASK DETAIL
https://phabricator.wikimedia.org/T262907
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, Svizel_pritula, CBogen, Akuckartz, darthmon_wmde, Nandana,
Namenlo
Mstyles added a comment.
Wrapping up the flink pipeline work, a patch is out for the integration tests
and there's some more test cases to be added for clarity on the Decide Mutation
Operation to ensure that all delete cases are covered. Pipeline consumer work
is up next
TASK DETAIL
Mstyles added a comment.
Clearing the state is out of the question, I think. Right now, we mark the
revision we see a delete at and then we don't allow a Rev Create Event.
However, this means that any regular RevCreate events don't allow the item to
be reimported ever again. I think
Mstyles added a comment.
What do we want to do if we get a PageDelete event for the same item multiple
times? Right now we're not tracking deletes, so I'm assuming multiple deletes
for the same item probably can't do that much harm.
TASK DETAIL
https://phabricator.wikimedia.org/T256875
Mstyles added a comment.
Shared model delete functionality is complete, currently in the process of
adding delete functionality to the flink pipeline
TASK DETAIL
https://phabricator.wikimedia.org/T256875
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
Mstyles removed Mstyles as the assignee of this task.
TASK DETAIL
https://phabricator.wikimedia.org/T251515
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Mstyles, dcausse, Zbyszko, Aklapper, Lea_Lacroix_WMDE, Gehel, CBogen,
Akuckartz
Mstyles added a project: Discovery-Search (Current work).
TASK DETAIL
https://phabricator.wikimedia.org/T256875
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, dcausse, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314
Mstyles claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T256875
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, dcausse, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314,
Lahi, Gq86
Mstyles closed subtask T251515: Automate data reload for SPARQL Endpoint for
Commons as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T251488
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Nintendofan885, Zbyszko, Smalyshev
Mstyles closed this task as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T251515
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Zbyszko, Aklapper, Lea_Lacroix_WMDE, Gehel, CBogen, Akuckartz,
darthmon_wmd
Mstyles claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T257314
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: CBogen, Lucas_Werkmeister_WMDE, Aklapper, Zbyszko, Akuckartz,
darthmon_wmde, Nandana, Namenlos314, Lahi
Mstyles added a comment.
Since the gui directory is overwritten by scap via symlinks, Guillaume
proposed that the config files live in /etc/config and that they be symlinked
to the gui directory. See more discussion
here:https://gerrit.wikimedia.org/r/c/operations/puppet/+/606297. I want
Mstyles added a project: Discovery-Search (Current work).
TASK DETAIL
https://phabricator.wikimedia.org/T251514
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: CBogen, Lucas_Werkmeister_WMDE, Zbyszko, WMDE-leszek, Addshore, Aklapper
Mstyles added a comment.
Here is the nginx example ->
https://github.com/nginxinc/NGINX-Demos/tree/master/oauth2-token-introspection-oss
TASK DETAIL
https://phabricator.wikimedia.org/T251500
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferen
Mstyles added a comment.
Some notes for when/if we do oauth
Docs -> https://www.mediawiki.org/wiki/OAuth/For_Developers
we might be able to use this npm package
<https://www.npmjs.com/package/passport-mediawiki-oauth> for auth with mediawiki
This
<https://github.com/SQL-en
Mstyles added a subscriber: Zbyszko.
Mstyles added a comment.
As discussed in email with @Zbyszko the script should do the following
1. Download the newest dump (maybe - we can provide it manually, but where's
the fun in that?)
2. Munge the data (running a script with appropriate data
Mstyles claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T251515
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Aklapper, Lea_Lacroix_WMDE, Gehel, CBogen, darthmon_wmde, Nandana, Lahi,
Gq86, Lucas_Werkmeister_WMDE
Mstyles closed this task as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T247123
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Gehel, Ottomata, Addshore, hashar, Aklapper, Jdforrester-WMF, Blissjay007,
Oblanc
Mstyles added a comment.
Docs are updated ->
https://wikitech.wikimedia.org/wiki/Wikidata_query_service#Production_Deployment
and verified that the zip file from the latest version can be downloaded from
Archiva
TASK DETAIL
https://phabricator.wikimedia.org/T247123
EMAIL PREFEREN
Mstyles added a subscriber: Gehel.
Mstyles added a comment.
Talked to @Gehel and the issue is probably that the archiva credentials that
come from analytics have different server ID's than the ones we use.
https://github.com/wikimedia/wikimedia-discovery-discovery-parent-pom/blob/master
Mstyles added a comment.
I know what that is, I forgot to add the archiva deployment profile. I'll put
a patch out
TASK DETAIL
https://phabricator.wikimedia.org/T247123
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Addshore, hashar
Mstyles added a comment.
@hashar the jenkins job failed due to no git auth to push. you can see it
here:
https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker-wdqs/3/
also is there a way to only have this job run when manually triggered and not
after every merge
Mstyles reopened this task as "Open".
TASK DETAIL
https://phabricator.wikimedia.org/T247123
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Addshore, hashar, Aklapper, Jdforrester-WMF, Blissjay007, Oblanco79,
Alter-paule,
Mstyles edited projects, added Discovery-Search (Current work); removed
Discovery-Search.
TASK DETAIL
https://phabricator.wikimedia.org/T247123
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Addshore, hashar, Aklapper, Jdforrester-WMF
Mstyles claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T247123
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Addshore, hashar, Aklapper, Jdforrester-WMF, Oblanco79, Alter-paule,
Beast1978, CBogen, Un1tY, Hook696
Mstyles added a comment.
@Jdforrester-WMF WMDE will be taking on responsibility for any new deployment
methods. That work will be tracked in T192006
<https://phabricator.wikimedia.org/T192006> and T210286
<https://phabricator.wikimedia.org/T210286>.
TASK DE
Mstyles added a comment.
@addshore that's correct, after removing the gui submodule, I won't be doing
any further work
TASK DETAIL
https://phabricator.wikimedia.org/T241291
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: WMDE-leszek
Mstyles added a comment.
After a bunch of discussion with the team, it's been decided that removing
the gui submodule from the RDF repository will suffice for now. That will fix
our broken build issues (see https://phabricator.wikimedia.org/T242640)
@Ladsgroup I definitely think you should
Mstyles added a comment.
@gehel I think we can consider this closed unless someone is able to reproduce
TASK DETAIL
https://phabricator.wikimedia.org/T232006
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Mstyles, Gehel, BBlack
Mstyles added a comment.
for clarification the correct response will contain a list that looks like
this
@prefix schema: <http://schema.org/> .
@prefix pq:<http://www.wikidata.org/prop/qualifier/> .
@prefix pr:<http://www.wikidata.org/prop/reference/>
Mstyles added a comment.
from inside any of the WDQS machines (
'wdqs1004.eqiad.wmnet','wdqs1005.eqiad.wmnet',
'wdqs1006.eqiad.wmnet','wdqs1007.eqiad.wmnet')
the following curls return the correct data
`curl localhost:80/bigdata/ldf` -> direct to nginx server on host
`curl localh
Mstyles added a project: Traffic.
Restricted Application added a project: Operations.
TASK DETAIL
https://phabricator.wikimedia.org/T237165
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles
Cc: Vahurzpu, Lucas_Werkmeister_WMDE, WikidataFacts
72 matches
Mail list logo
