Re: [Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey folks!

I should really stop calling this a weekly update because it's getting a
bit silly at this point.  :)   But if it were a weekly update, it would
cover the weeks of 42 - 46.

*Highlights:*

   - 3 new models: Finnish Wikipedia (reverted) and Estonian Wikipedia
   (damaging & goodfaith)


   - We estimated and agreed on funding for ORES servers in the next year
   with Operations


   - We published a paper about vandalism detection in Wikidata and a blog
   post about the massive effect of some initiatives on coverage of Women
   Scientists in Wikipedia.


*New development:*

   - We added recall-based threshold metrics to the new draftquality model
   which should help tool devs know what which new page creations to highlight
   for review[1]


   - We added optional notices for ORES pages which will help us visually
   distinguish our experimental install in WMFlabs from the Prod install (
   ores.wikimedia.org)[2]


   - We added basic language support for Finish (Thanks 4shadoww)[3] and
   deployed a 'reverted' model[4]


   - We lead a discussion in Wikidata about "item quality" that resulted in
   a Wikipedia 1.0 like scale for Wikidata quality[5,6] and designed a
   Wikilabels form to capture the gist of it[7]


   - We enabled the ORES Review Tool on Czech Wikipedia[8]


   - We configured ChangeProp to use our new minified JSON output to save
   bandwidth[9]


   - We extended the Estonian language assets (Thanks Cumbril)[10] and
   deployed the 'damaging' and 'goodfaith' models[11,12]


   - We enabled a testing model for 'goodfaith' on the Beta Cluster to make
   it easier for the Collaboration team to run tests with their new filter
   interface[13]


   - We created a new "precache" endpoint that will allow us to
   de-duplicate configuration with ChangeProp and handle all routing in ORES
   locally[14]


*Resourcing:*

   - We completed a 2 year estimate of ORES resource needs and discussed
   funding (capital expendature) for ORES in the coming fiscal year[15].  This
   will allow us to continue to grow ORES both in number of models and in
   scoring capacity.


*Communications:*

   - Amir improved the KDD paper based on review feedback[16] and got it
   published[17]


   - We published a blob post about our measurements of WikiProject Women
   Scientists[18,19] -- "The Keilana Effect"


   - Thanks to Cumbril's work, the Estonian labeling campaing was
   finished[20]


*Deployments:*

   - In early February, we deployed a new set of translations to Wikilabels
   (specifcally targeting Romanian Wikipedia)[21]


   - In mid-February, we deployed some fixes to ORES documentation and
   response formatting[22]


   - In mid-March, we deployed 3 new scoring models and ORES notices[23]


*Maintenance and robustness:*

   - We fixed a serious issue in the "mwoauth" library that Wikilabels
   depends on[24]


   - We reduced the number of revisions per request that we could receive
   via api.php[25]


   - We investigated a scap issue that broke ORES deployment[26]


   - We fixed a minor issue with JSON minification behavior[27] and
   hard-coding of the location of ORES in the documentation[28]


   - We improved performance of ORES filters on MediaWiki[29]


   - We improved the language describing ORES behavior on
   Special:Contributions[30]


   - We added a notice to the Wikipages that Dexbot maintains about its
   behavior[31]


   - We added notices to ores.wmflabs.org about it's experimental nature[32]


   - We fixed some issues with testing Finnish language assets[33]


   - We fixed some styling issues that resulted from an upgrade of OOJS
   UI[34]


1. https://phabricator.wikimedia.org/T157454 -- Add recall based thresholds
to draftquality model
2. https://phabricator.wikimedia.org/T150962 -- Add an optional notice to
ORES main and ui pages
3. https://phabricator.wikimedia.org/T158587 -- Add language support for
Finnish
4. https://phabricator.wikimedia.org/T160228 -- Train/test reverted model
for fiwiki
5. https://phabricator.wikimedia.org/T157489 -- [Discuss] item quality in
Wikidata
6. https://www.wikidata.org/wiki/Wikidata:Item_quality
7. https://phabricator.wikimedia.org/T155828 -- Design item_quality form
for Wikidata
8. https://phabricator.wikimedia.org/T151611 -- Enable ORES Review Tool on
Czech Wikipedia
9. https://phabricator.wikimedia.org/T157693 -- Use minified JSON format in
ChangeProp
10. https://phabricator.wikimedia.org/T160193 -- Extend estonian language
assets from Wiki page
11. https://phabricator.wikimedia.org/T159608 -- Train/test
damaging/goodfaith models for etwiki
12. https://phabricator.wikimedia.org/T130280 -- Deploy edit quality models
for etwiki
13. https://phabricator.wikimedia.org/T160467 -- Enable 'goodfaith' on
testwiki on Beta Cluster
14. https://phabricator.wikimedia.org/T148714 -- Create generalized
"precache" endpoint for ORES
15. https://phabricator.wikimedia.org/T157222 -- Estimate ORES capex for
FY2017-18
16. 

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey folks!

This is the 32 - 41st weekly update from the revision scoring team that we
have sent to this mailing list.  We've been busy, but our reporting fell
behind.  So here I am getting us caught up!  This is going to be a long
one.  Bear with me.

One major thing we've done in the past few weeks is drafted and presented a
proposal to increase the resourcing for the ORES project in the 2017 Fiscal
Year.  Currently, we're just one fully funded staff member (halfak) and
partially funded contractor (Amir1) working with a bunch of volunteers.
We're proposing to staff the team with fulltime engineers, a liaison and a
tech writer.  See a full draft of our proposal and pitch deck here:
https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Scoring_Platform_team

*New development:*

We've expanded support for our "editquality" models to more wikis and
improved the performance of some of the models.

   - We scaled up the number of observations for Indonesian Wikipedia to
   100k[1]


   - We added language support for Romanian[2] and built the basic
   "reverted" model[3]


   - We trained and tested "damaging" and "goodfaith" models for Czech
   Wikipedia[4]


   - We implemented some params in our training utilites to control memory
   usage[5]


   - We deployed all of the above to Wikimedia Labs[6].  A production
   deployment is coming soon.


Prompted by the 2016 community wishlist[7], we've implemented a
"draftquality" model for evaluating new page creations.

   - We researched deletion reasons on English Wikipedia[8] and created a
   labeled dataset using the deletion log.


   - We engineered a set of features to predict the quality of new
   articles[9] and built a model[10]


   - We generated a set of datasets[11,12,13] to make it easier for
   volunteers and external researchers to help us audit the performance of the
   model.


   - We deployed the model on WMFLabs[14] and announced it's presence to a
   few interested patrollers in English Wikipedia


   - We've started the process of deploying the model in production[15,16]


We completed a project exploring the use of advance natural-language
processing strategies to extract new signal about vandalism, article
quality and problematic new articles.  Regretfully, memory issues prevent
us from trivially putting this into production[17], so we're looking into
alternative strategies[18].

   - We implemented a strategy for extracting sentence from Wikitext[19]


   - We built sentence banks for personal attacks[20, vandalism[21],
   spam[22], and Featured Articles[23].


   - We built PCFG-based models[24] and analyzed their ability to
   differentiate[25]


We've been working with the Collaboration Team[26] on their Edit Review
Improvments project[27]

   - We defined and implemented a set of new precision-based test
   statistics that will inform thresholds used in their new user interface[28]


   - But we also decided to continue to report recall-based test statistics
   as well[29]


Based on advice from engineers on the Collaboration Team, we've begun the
process of converting Wiki labels[30] to a stand-alone tool in labs.

   - We generalize the gadget interface so that it can handle all
   langauges/wikis[31]


   - We implemented a means to auto-configure wikis based on the
   dbname[32,33] and that allowed us to simplify configuration[34]


   - We also implemented some performance improvements with minification,
   bundling[35]


*Labeling:*

In the past few weeks, we've set up labeling campaigns for a few wikis.

   - We deployed an edit types campaign for Catalan Wikipedia[36]


   - We deployed an edit quality campagin for Chinese[37] and Romanian[38]
   Wikipedias


   - We deployed a new type of campaign for English Wikipedia --
   "discussion quality" asks editors to label talk posts as "toxic" or not[39]


*Maintenance and robustness:*

We've solved a large set of problems with logging issues, compatibility
with wikibase, and we've made minor improvements to performance.

   - We addressed a few bugs in the ORES Review Tool[40,44]


   - We quieted some errors from our logging in ORES[41,45]


   - We updated our code to work with a wikibase schema change[42]


   - We fixed a language fallback pattern in Wiki labels[43]


   - We set up monitoring on ORES database disk sizes[46]


   - We fixed some issues with scap, phabricator's diffusion and other
   supporting systems so that we can continue deploying to beta labs[47]


   - We split our assets repo so that we can let our WMFLabs deploy get
   ahead of the Production deployment[48]


   - ORES can now minify its JSON responses[49]


   - We identified a bug in flask-assets and worked around it in our local
   installation of Wiki labels[50]


*Communications and outreach:*

We had a big presence at the Wikimedia Developer summit, we've drafted a
resourcing proposal, and we've made some announcements about upcoming plans
for the ORES Review tool.

   - We facilitated the 

[Wikitech-l] The Revision scoring weekly update

[Aaron Halfaker]

Hey,

This is the 30th and 31st weekly update from the revision scoring team that
we have sent to this mailing list.  We accidentally skipped a week again.

*New development:*

   - We added a new "lowest" sensitivity level to ORES review tool.  This
   new sensistivity level will only flag edits that ORES is very confident are
   actually damaging[1].


   - We applied the MediaWiki standard color palette to Wikilabels[2]


   - We generated a manually censored public dataset of
   spam/vandalism/attack pages[3].  This will help others to develop spam,
   vandalism and attack page detection models.  See the publication of the
   dataset[4].


   - We've implement color-based confidence reporting for ORES damage
   detection[5]


*Maintenance and robustness:*

   - We updated the version of OOjs-UI that gets bundled with Wiki
   labels[6] and moved the static assets to a new repositiory[7]


   - We fixed an issue in the recscoring library[8] that caused ORES to
   return invalid JSON and rendered the UI useless[9].


*Communications:*

   - We gave a 3 minute presentation on the state of ORES to Victoria
   Coleman, the WMF's new CTO[10].


   - We performed a basic analysis of Wikipedia article quality trends
   using the dataset we released a few weeks ago[11].  We'll have a more
   substantial analysis soon.


   - We made a post on the ORES review tool talk page[12,13] detailing how
   we plan to incorporate a new filtering strategy into the ORES review tool.
   Please join the discussion there.


1. https://phabricator.wikimedia.org/T150224 -- Add "Lowest" ORES
sensitivity for fpr=0.1
2. https://phabricator.wikimedia.org/T151119 -- Apply ui standardization
color palette to Wikilabels
3. https://phabricator.wikimedia.org/T150307 -- Create manually vetted
dataset of spam/vandalism/attack pages
4. https://dx.doi.org/10.6084/m9.figshare.4245035
5. https://phabricator.wikimedia.org/T144922 -- Visually report damaging
confidence
6. https://phabricator.wikimedia.org/T151222 -- Update bundled OOJS-ui with
Wikilabels
7. https://github.com/wiki-ai/flask-oojsui
8. https://phabricator.wikimedia.org/T150961 -- ORES ui is broken (text
field disabled)
9. https://github.com/wiki-ai/ores/issues/177
10. https://phabricator.wikimedia.org/T150544 -- ORES (a 2-3 minute
presentation)
11. https://phabricator.wikimedia.org/T151214 -- Basic analysis of
Wikipedia quality using monthly predictions
12. https://phabricator.wikimedia.org/T150858 -- Post about ORES review
tool including ERI filters
13. https://www.mediawiki.org/wiki/Topic:Tflhjj5x1numzg67

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision scoring weekly update

[Aaron Halfaker]

Hey,

This is the 29th weekly update from revision scoring team that we have sent
to this mailing list.

Deployments:

   - We deployed logging changes to ORES that will reduce the verbosity[1]


   - We also deployed revscoring 1.3.0 and new models built with it to WMF
   labs[2].  This won't change anything important from a user-perspective, but
   it paves the way for developing new modeling strategies.


Maintenance and robustness:

   - We fixed puppet so that log file directories are also created on the
   celery worker nodes (affects wmflabs)[3]


   - We fixed an issue with our recall_at_fpr metrics which was incorrectly
   defined and implemented a recall_at_precision metric to take its place[4]


New development:

   - We've made a lot of progress on modeling sentences and have just
   started experimenting with a sentence model from featured articles[5]


   - We're reviewing a dataset of spam/vandalism/attack new page creations
   for public release[6].  This dataset will help our collaborators work with
   us on modeling the quality of drafts and supporting new page triage.


1. https://phabricator.wikimedia.org/T149730 -- Deploy logging changes to
ORES
2. https://phabricator.wikimedia.org/T150447 -- Deploy revscoring 1.3.0 and
updated editquality and wikiclass to wmflabs
3. https://phabricator.wikimedia.org/T149925 -- /srv/log/ores/ not created
on worker nodes
4. https://phabricator.wikimedia.org/T149825 -- Implement recall at
precision (and fix FPR metrics)
5. https://phabricator.wikimedia.org/T148867 -- Implement sentences
datascources & experiment with normalization.
6. https://phabricator.wikimedia.org/T150307 -- Create manually vetted
dataset of spam/vandalism/attack pages

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision scoring weekly update

[Amir Ladsgroup]

Hey,
This is 28th weekly update from revision scoring team that we have sent to
this mailing list.

Publications:

   - *New dataset shows fifteen years of Wikipedia’s quality trends *posted
   in Wikimedia Blog [1,2]


   - Halfaker, Aaron (2016): Monthly Wikipedia article quality predictions.
   *figshare*. doi.org/10.6084/m9.figshare.3859800 [3,4]


Maintenance and robustness:

   - Now, most of ORES extension source code is covered by continuous
   integration tests [5]


   - In order to keep track of changes in ORES grafana dashboards, we keep
   their JSON content in github now [6]


   - Implemented new metric for grafana: datasources_extracted [7]


New development

   - Thanks to the reading team, Now ORES extension has API modules to
   expose ORES scores, [8] filter on recent changes and watchlist, [9] and
   exposing ores models data [10]


1. https://blog.wikimedia.org/2016/10/27/wikipedia-quality-trends-dataset/
2. https://phabricator.wikimedia.org/T146709
3. https://dx.doi.org/10.6084/m9.figshare.3859800
4. https://phabricator.wikimedia.org/T145332
5. https://phabricator.wikimedia.org/T146560
6. https://phabricator.wikimedia.org/T149347
7. https://phabricator.wikimedia.org/T149199
8. https://phabricator.wikimedia.org/T143614
9. https://phabricator.wikimedia.org/T143616
10.https://phabricator.wikimedia.org/T143617

Sincerely,
Amir from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 26th and 27th weekly update from revision scoring team that we
have sent to this mailing list. We forgot to send the update for last week!

Last week, we were featured in Research's quarterly review. In the last 3
months, we achieved our goals to expand the ORES extension to 6 wikis (we
made it to 8!) and to release datasets of article quality predictions. The
minutes from the quarterly review are not yet online, but once they are,
you'll be able to see them at [1].

Maintenance and robustness:

   - We discussed and decided on a set of strategies for handling
   goodfaith/naive DOS attacks on ORES[2]


   - We fixed an i18n issue in Wiki Labels[3]


   - We updated the article quality models (wikiclass/wp10) to use
   revscoring 1.3.0[4]


   - We investigated and solved a memory leak in our pre-caching utility[5]


   - We configured celery to send its logs to a place where we can read
   them for easier debugging[6]


   - We deployed a set of schema changes to constrain the ORES Review Tools
   database appropriately[7]


   - Also worth noting is that the services cluster (SCB) has been
   expanded[8].  ORES has now doubled in capacity


Datasets

   - We discussed how to make the historical article quality dataset
   available via quarry[8]. Regretfully, it seems that we'll not be able to do
   that for at least a couple of months.


New development

   - We've implemented embedding of machine-readable scores in a JS
   variable on-wiki[9]. This will make it easier for tool developers to
   experiment with new ways of displaying Special:RecentChanges more easily.
   It's also a necessary precondition for adding color-based signaling of
   ORES' confidence about an edit.


1.
https://meta.wikimedia.org/wiki/Wikimedia_Foundation_metrics_and_activities_meetings/Quarterly_reviews/Research,_Design_Research,_Analytics,_and_Performance,_October_2016
2. https://phabricator.wikimedia.org/T148347 -- [Discuss] DOS attacks on
ORES. What to do?
3. https://phabricator.wikimedia.org/T139587 -- Revision not found error
unformatted and not localized
4. https://phabricator.wikimedia.org/T147201 -- Update wikiclass for
revscoring 1.3.0
5. https://phabricator.wikimedia.org/T146500 -- Investigate memory leak in
precached
6. https://phabricator.wikimedia.org/T147898 -- Send celery logs to
/srv/log/ores instead of /var/lib/daemon.log
7. https://phabricator.wikimedia.org/T147734 -- Review and deploy 309825
8. https://phabricator.wikimedia.org/T147903 -- Expand SCB cluster
9. https://phabricator.wikimedia.org/T146718 -- [Discuss] Hosting the
monthly article quality dataset on labsDB
10. https://phabricator.wikimedia.org/T143611 -- Embed machine readable
ores scores as data on pages where ORES scores things

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 24th and 25th weekly update from revision scoring team that we
have sent to this mailing list. We skipped a week due to travel and other
work.

Maintenance and robustness:

   - We improved the performance of RecentChanges fitlering in the ORES
   extension[1]


   - We built and ran a maintenance script to clean up duplicate cached
   data for the ORES extension[2,3]


   - We updated the editquality models for the new version of revscoring
   (1.3.0)[4] and made some upstream changes to json2tsv to make that easier[5]


   - We quited down some of our error reporting so that our logs take up
   less space[6]


Datasets:

   - We generated a dataset that uses the "wp10" prediction model to assess
   article quality in monthly intervals for English, French, and Russian
   Wikipedia[7].  This should enable new research into the quality dynamics of
   these wikis.


   - We generated a dataset of vandalism, spam, and attack page creations
   for building a new "draft quality" model[8]


Communication:

   - Presented about transparent/open AI development practices around ORES
   at the Association of Internet Researchers[9]


New development:

   - We've made substantial progress towards adding ORES data to
   MediaWiki's api.php endpoints with rcshow=oresreview[10] and rvprop=ores[11]


1. https://phabricator.wikimedia.org/T146111 -- hidenondamaging=1 query is
extremely slow on enwiki
2. https://phabricator.wikimedia.org/T145356 -- Ensure ORES data violating
constraints do not affect production
3. https://phabricator.wikimedia.org/T145503 -- Build a maintenance script
to clean up duplicate data
4. https://phabricator.wikimedia.org/T146410 -- Update editquality for
revscoring 1.3.0
5. https://phabricator.wikimedia.org/T146939 -- Add type decoding support
to tsv2json
6. https://phabricator.wikimedia.org/T146680 -- Quiet result.get Warning in
tasks
7. https://phabricator.wikimedia.org/T145655 -- Generate monthly article
quality dataset
8. https://phabricator.wikimedia.org/T135644 -- Generate spam and vandalism
new page creation dataset
9. https://phabricator.wikimedia.org/T147706 -- Present about ORES
transparency at AoIR
10. https://phabricator.wikimedia.org/T143616 -- Introduce
rcshow=oresreview and similar ones
11. https://phabricator.wikimedia.org/T143614 -- Introduce ORES rvprop

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 23rd weekly update from revision scoring team that we have sent
to this mailing list.

New development

   - We implemented and demonstrated a linguistic/stylometric processing
   strategy that should give us more signal for finding vandalism and
   spam[1].  See the discussion on the AI list[2].


   - As part of our support for the Collaboration Team, we've been
   producing tables of model statistics that correspond to set of
   thresholds[3].  This helps their designers work on strategies for reporting
   prediction confidence in an intuitive way.


Maintenance and robustness

   - We had a major downtime event that was caused by our logs being too
   verbose.  We've recovered and turned down the log level[4].


   - We made sure that halfak got pings when ores.wikimedia.org goes down[5]


Datasets

   - We created a database on Wikimedia Labs that provides access to a
   dataset containing a complete set of article quality predictions for
   English Wikipedia[6].  See our announcements[7,8,9].


1. https://phabricator.wikimedia.org/T146335 -- Implement a basic scoring
strategy for PCFGs
2. https://lists.wikimedia.org/pipermail/ai/2016-September/98.html
3. https://phabricator.wikimedia.org/T146280 -- Produce tables of stats for
damaging and goodfaith models
4. https://phabricator.wikimedia.org/T146581 -- celery log level is INFO
causing disruption on ORES service
5. https://phabricator.wikimedia.org/T146720 -- Ensure that halfak gets
emails when ores.wikimedia.org goes down
6. https://phabricator.wikimedia.org/T106278 -- Setup a db on labsdb for
article quality that is publicly accessible
7. https://phabricator.wikimedia.org/T146156 -- Announce article quality
database in labsdb
8. https://lists.wikimedia.org/pipermail/ai/2016-September/91.html
9.
https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)/Archive_149#ORES_article_quality_data_as_a_database_table

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 22nd weekly update from revision scoring team that we have sent
to this mailing list.

UI work:

   - We configured the default threshold for the ORES review tool on
   Wikidata to be more strict (higher recall, lower precision)[1]


   - We fixed a display issue on Special:Contributions where the filters
   would not wrap[2]


Increasing model fitness:

   - We finished demonstrating model fitness gains using hash-vector
   features[3].  Next, we'll be working to get the hash-vector features
   implemented in revscoring/ORES[4].


   - We implemented a new strategy for training and testing on all data
   using cross-validation[5].  This will both increase the fitness of the
   models and make the statistics reported more robust.


Maintenance and robustness

   - We fixed an indexing issues in ores_model that prevented the
   deployment of updated models[6].


   - We did a minor investigation to a short period of degraded service
   quality on WMF Labs[7]


1. https://phabricator.wikimedia.org/T144784 -- Change default threshold
for Wikidata to high
2. https://phabricator.wikimedia.org/T143518 -- Filter on user contribs has
nowrap, causing issues
3. https://phabricator.wikimedia.org/T128087 -- [Spike] Investigate
HashingVectorizer
4. https://phabricator.wikimedia.org/T145812 -- Implement ~100 most
important hash vector features in editquality models
5. https://phabricator.wikimedia.org/T142953 -- Train on all data, Report
test statistics on cross-validation
6. https://phabricator.wikimedia.org/T144432 -- oresm_model index should
not be unique
7. https://phabricator.wikimedia.org/T145353 -- Investigate short period of
ores-web-03 insanity

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 21st weekly update from revision scoring team that we have sent
to this mailing list.

New development

   - We received a request to get moving on Spanish Wikibooks support, so
   we dug in:


   - We deployed a new Wiki labels campaign[1]


   - We fixed an issue in Wiki labels that prevented requests from *.
   wikibooks.org[2]


   - We trained a basic "revert" detection model that seems to be pretty
   effective[3]


   - We also generated a dataset of article quality scores for English
   Wikipedia[4].  You can download it here: [5]


This week, we invested in some long term tasks.  If you review our
phabricator board, you'll see substantial progress in improving our damage
detection models with hashing vectorization strategies[6, 7], implementing
a more robust model testing strategy[8], and implementing some advance
natural language processing strategies[9, 10].  Stay tuned for the
completion of these activities in the coming weeks.

1. https://phabricator.wikimedia.org/T143962 -- Add uniqueness constraints
to ores_classification
2. https://phabricator.wikimedia.org/T145406 -- Fix CORS for wikibooks
3. https://phabricator.wikimedia.org/T145428 -- Train/test reverted model
for Spanish Wikibooks
4. https://phabricator.wikimedia.org/T135684 -- Generate recent article
quality scores for English Wikipedia
5.
https://datasets.wikimedia.org/public-datasets/enwiki/article_quality/wp10-scores-enwiki-20160820.tsv.bz2
6. https://phabricator.wikimedia.org/T128087 -- [Spike] Investigate
HashingVectorizer
7. https://en.wikipedia.org/wiki/Feature_hashing
8. https://phabricator.wikimedia.org/T142953 -- Train on all data, Report
test statistics on cross-validation
9. https://phabricator.wikimedia.org/T144636 -- Implement PCFG features
10. https://en.wikipedia.org/wiki/Stochastic_context-free_grammar

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 20th weekly update from revision scoring team that we have sent
to this mailing list.

New development:

   - We implemented the basic functionality for handling bag of words and
   other types of abstract feature vectors in `revscoring`. [1] This required
   some changes to some dependencies as well. [2]


   - We extended the user-group related features to include more of the
   dominant groups outside of English Wikipedia [3] and incremented the models
   that changed substantially [4]


Documentation:

   - We extended the documentation at mw:Extension:ORES to make it easier
   for new developers to work with us. [5]


Resourcing:

   - We discussed the teams resourcing needs (hardware, engineering, and
   community liaison support) with Wes Moran. [6]


Maintenance and robustness:

   - We addressed a variety of issues around caching and how the ORES
   extension loads new data


   - ORES now returns headers that will disable secondary caching. [7]


   - Our maintenance scripts will circumvent caches that do not listen to
   no-cache headers. [8, 9]


   - We fixed an issue where the ORES review tool would duplicate items in
   Special:RecentChanges. [10]


   - We standardized the extraction pattern for the enwiktionary model so
   that it looks similar to other models. [11]


1. https://phabricator.wikimedia.org/T132580 -- Implement abstraction for
Sparse Feature Vectors
2. https://phabricator.wikimedia.org/T144430 -- Update yamlconf so that
import_path can handle deep attributes
3. https://phabricator.wikimedia.org/T143909 -- Extend user group features
4. https://phabricator.wikimedia.org/T144855 -- Increment ruwiki
editquality models
5. https://phabricator.wikimedia.org/T144676 -- Improve technical
documentation in Extension:ORES in mediawiki.ore
6. https://phabricator.wikimedia.org/T144517 -- ORES and Product:
resourcing discussion
7. https://phabricator.wikimedia.org/T144193 -- Set max-age header to 0
seconds for ORES to quiet secondary caches
8. https://phabricator.wikimedia.org/T144196 -- Get model version needs to
invalidate cache
9. https://phabricator.wikimedia.org/T144195 -- Check model version
replaces every time it runs.
10. https://phabricator.wikimedia.org/T144233 -- Redundant results in ORES
review tool
11. https://phabricator.wikimedia.org/T144605 -- Fix makefile entry for
enwiktionary.rev_reverted.20k_2016.tsv

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 19th weekly update from revision scoring team that we have sent
to this mailing list.

Deployments:

   - We deployed a set of new models to ORES that reduce our memory usage
   and slightly increase fitness. [1]  These models were discussed in an email
   to the "ai" mailing list. [2]


   - We also completed a major quarterly goal.  The ORES review tool is now
   deployed as a beta feature on 8 wikis! [3]  This came with some quick fixes
   to fix some confusion and usability issues. [4]  The beta feature is now
   available on English, Polish, Portuguese, Russian, Dutch, Persian and
   Turkish Wikipedias as well as Wikidata.


New development:

   - We discussed and came to a rough consensus about how to integrate ORES
   into api.php. [5]


   - We deployed a new edit quality campaign on English Wikipedia to gather
   more data for training ORES. [6, 7]


   - We added a specific set of user groups to the ORES models for Turkish
   Wikipedia and saw an increase in model fitness. [8]


Maintenance and robustness:

   - We fixed bugs in our maintenance scripts for purging old model
   versions [9, 10]


   - We switch to using our production models on the beta labs cluster so
   now we can catch vandalism there too (and know that the models actually
   work) [11]


   - We improved the error messages reported from Wiki Labels so that the
   actual error appears when the API responds with non-200 HTTP status [12]


1. https://phabricator.wikimedia.org/T144101 -- Deploy ORES at 2016-08-29
2. https://lists.wikimedia.org/pipermail/ai/2016-August/68.html
3. https://phabricator.wikimedia.org/T140002 -- [Epic] Deploy ORES review
tool
4. https://phabricator.wikimedia.org/T143988 -- $wgOresModels set all
models true
5. https://phabricator.wikimedia.org/T122689 -- [Discuss] api.php
integration with ORES
6. https://phabricator.wikimedia.org/T143745 -- Deploy 2016 edit quality
campaign to English Wikipedia
7. https://en.wikipedia.org/wiki/Wikipedia:Labels/Edit_quality
8. https://phabricator.wikimedia.org/T140474 -- Include specific user
groups in the trwiki edit quality model
9. https://phabricator.wikimedia.org/T144216 -- Purge model score should
clean when there is no row is ores_model too
10. https://phabricator.wikimedia.org/T143798 -- Update model versions is
badly broken in ORES extension
11. https://phabricator.wikimedia.org/T143567 -- Switch beta to use the
proper wiki models for scoring (rather than "testwiki")
12. https://phabricator.wikimedia.org/T138255 -- Wikilabels UI reports
non-200 status errors badly

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 18th weekly update from revision scoring team that we have sent
to this mailing list.

*Communications:*

   - Aaron presented on how user-feedback has been helping us address some
   sneaky biases in ORES' models. [1, 2, 3]


*New development:*

   - We included 'autoreview' and 'patroller' groups in Turkish wiki models
   to get a fitness boost. [4]


   - We added some basic uwsgi metrics to grafana[5] and added a response
   timing metric from Change Propagation so that we can track any performance
   issues. [6]


*Maintenance and robustness:*

   - We increased the number of workers per node in production for a 66%
   increase in total capacity for ORES[7]


   - We updated all of our edit quality models with the new version of
   revscoring [8] and sent an email out to wikitech-l and ai-l about the
   implications for tool developers. [9]


   - We decided not to make specialized models for ORES in beta labs. [10]
   Instead, we'll use the production models so that issues with them will be
   caught in beta.


1. https://phabricator.wikimedia.org/T143275 -- Present on user-feedback
stories at Research Showcase
2. https://www.youtube.com/watch?v=rsFmqYxtt9w#t=29m00s -- Video of ORES
user-feedback talk
3.
https://www.mediawiki.org/wiki/File:Deploying_and_maintaining_AI_in_a_socio-technical_system_--_Research_Showcase_(August_2016).pdf
4. https://phabricator.wikimedia.org/T140474 -- Include specific user
groups in the trwiki edit quality model
5. https://phabricator.wikimedia.org/T143081 -- Add uwsgi-related metrics
to grafana
6. https://phabricator.wikimedia.org/T143568 -- Add median, 75% and 95%
response time to ORES dashboard
7. https://phabricator.wikimedia.org/T143105 -- Increase celery workers to
40 per scb node
8. https://phabricator.wikimedia.org/T143125 -- Update editquality models
with new version of revscoring
9. https://lists.wikimedia.org/pipermail/ai/2016-August/68.html --
"[AI] New models coming to ORES & notes"
10. https://phabricator.wikimedia.org/T141980 -- Should we make a model for
ores in beta?

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Amir Ladsgroup]

Hey,

This is the 17th weekly update from revision scoring team that we have sent
to this mailing list.

New developments:

   - ORES review tool is now deployed in Polish Wikipedia [1]


   - ORES review tool shows "r" flag in user contributions too. This will
   be deployed this week [2]


   - We increased the number of workers which increases ORES capacity to
   generate scores [3]


   - Revscoring now supports Tamil language. [4]


   - Our precaching now produces new metrics like failure rate and scoring
   speed [5]


Maintenance and robustness:

   - ORES extension now just marks jobs as failed instead of throwing
   exception and spamming in errors log when it can't score an edit. [6]


   - We changed the way our precaching works causing a huge reduction in
   CPU usage. [7]


   - Fixed issue with ORES review tool and flow board activity. [8]


   - There was some inconsistency in the API scheme in before and after the
   deployment. We fixed it with another deployment [9]


   - We made our deploy process more robust by switching our canary node in
   codfw for on in eqiad [10]


   - We migrated our wp10 models from Random Forest to Gradient Boosting.
   It gives us almost the same accuracy with a reduction in memory usage. [11]


   - We enabled uwsgi metrics for ores. It adds a lots of useful metrics
   such as average response time or active workers, etc. [12]


1. https://phabricator.wikimedia.org/T140005
2. https://phabricator.wikimedia.org/T132371
3. https://phabricator.wikimedia.org/T142361
4. https://phabricator.wikimedia.org/T134105
5. https://phabricator.wikimedia.org/T119341
6. https://phabricator.wikimedia.org/T141978
7. https://phabricator.wikimedia.org/T142360
8. https://phabricator.wikimedia.org/T142858
9. https://phabricator.wikimedia.org/T142857
10. https://phabricator.wikimedia.org/T142630
11. https://phabricator.wikimedia.org/T141603
12. https://phabricator.wikimedia.org/T141543

Sincerely,
Amir from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey folks,

This is the 16th weekly update from revision scoring team that we have sent
to this mailing list.

New developments:

   - We created dashboards for the ORES service in the Beta cluster[1] and
   created panes for tracking failed jobs[2].


   - We extended the documentation for the ORES review tool[3,4]


Maintenance:

   - We did some work to make the Beta cluster look more like production so
   that we can do better testing before the next deployment


   - We set up a password on the Beta redis server[5]


   - We configured the Beta ORES extension to actually use the Beta ORES
   service[6]


   - We also prepared a set of puppet changes for the deployment of a
   refactored version of ORES to production[7]


Issues in WMFLabs

   - We investigated a series of timeout errors that were appearing in the
   logs[8]


   - We investigated a periodic redis-related error that shower up when
   scoring edits[9]


   - We fixed our "05" web node that was periodically running out of
   memory[10]


Estimating future resource needs

   - In preparation for buying new hardware, we measured our past memory
   usage and extrapolated forward two years to estimate what hardware
   requirements we'll have[11]



   1. https://phabricator.wikimedia.org/T142294 - Dashboard or pane for
   ORES service in beta


   1. https://phabricator.wikimedia.org/T142119 - Dashboard or pane for
   ORES failed jobs on beta


   1. https://phabricator.wikimedia.org/T140150 - Make user-centered
   documentation for review tool


   1. https://www.mediawiki.org/wiki/ORES_review_tool


   1. https://phabricator.wikimedia.org/T141823 - Set up password on ORES
   Beta redis server


   1. https://phabricator.wikimedia.org/T141825 - Config beta ORES
   extension to use the beta ORES service


   1. https://phabricator.wikimedia.org/T141575 - Puppet config changes for
   ORES refactor


   1. https://phabricator.wikimedia.org/T141368 - [Investigate] ORES time
   out errors in logs


   1. https://phabricator.wikimedia.org/T141946 - [Investigate] Periodic
   redis related errors in wmflabs


   1. https://phabricator.wikimedia.org/T141523 - [Investigate] web-05
   downtime


   1. https://phabricator.wikimedia.org/T142046 - Extrapolate memory usage
   per worker forward 2 years


Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hey,

This is the 15th weekly update from revision scoring team that we have sent
to this mailing list.

*New developments:*

   - We'll no longer unnecessarily load the models into memory on the web
   workers[1].


   - We can now score multiple models against the same revision ID for
   (essentially) free[2].


   - Our precaching system will take advantage of this to drop load by
   about 3X[3].


   - Update wmflabs deploy repo for new version of ORES[4].


*Documentation & maintenance:*

   - We completed deployment and maintenance docs for Wiki labels[5], which
   means we've now got complete docs for our systems[6].


   - We implemented basic continuous integration tests for the ORES
   extension[7].


*Downtime:*

   - We had a 1 hour long downtime while trying to deploy new code to
   ores.wikimedia.org[8].  We've filed two critical tasks for making sure
   we don't make the mistake again[9,10].


1. https://phabricator.wikimedia.org/T134606 - Score multiple models with
the same cached dependencies
2. https://phabricator.wikimedia.org/T139407 - Don't load models into
memory of web workers
3. https://phabricator.wikimedia.org/T141376 - Update precached to group
requests by model
4. https://phabricator.wikimedia.org/T141377 - Update wmflabs deploy repo
for new version of ORES
5. https://phabricator.wikimedia.org/T131768 - Wikilabels deployment docs
6. https://phabricator.wikimedia.org/T106271 - Document maintenance tasks
7. https://phabricator.wikimedia.org/T140455 - CI test for ORES extension
8. https://wikitech.wikimedia.org/wiki/Incident_documentation/20160801-ORES
9. https://phabricator.wikimedia.org/T141823 - Set up password on ORES Beta
redis server
10. https://phabricator.wikimedia.org/T141825 - Config beta ORES extension
to use the beta ORES service

Sincerely,
Aaron from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Amir Ladsgroup]

Hey,
This is the 14th weekly update from revision scoring team that we have sent
to this mailing list.

New developments:

   - We have a new grafana dashboard for requests that the ORES beta
   feature makes [1] [2]


   - There is a grafana chart in ORES service showing total number of
   requests (including requests hitting cache). Showing we had 20 million
   requests last month! [3]


   - ORES extension can be set to score edits only in a given set of
   namespaces. This was useful to reduce failure rate of jobs in ORES
   extension from 4% to 1% [4]


   - Scap3 now automatically deploys to canary node for ORES first (it's
   scb2001.codfw.wmnet). So we can better test deployments beforehand. [5]


Maintenance and robustness:

   - We are removing a redundant index on ores_classification. No harm
   would be made but we'll save disk space. [6]


   - In footer of ores.wikimedia.org, we had "Hosted in Wikimedia Labs". It
   got fixed now. [7]


   - Cache.php in ORES extension which stores the scores into database has
   a better structure for handling errors.[8]


1. https://phabricator.wikimedia.org/T141169
2. https://grafana.wikimedia.org/dashboard/db/ores-extension
3. https://grafana.wikimedia.org/dashboard/db/ores
4. https://phabricator.wikimedia.org/T139660
5. https://phabricator.wikimedia.org/T139825
6. https://phabricator.wikimedia.org/T140803
7. https://phabricator.wikimedia.org/T139028
8. https://phabricator.wikimedia.org/T137880

Sincerely,
Amir from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Amir Ladsgroup]

Hey,
This is the (12 + 1)th [1] weekly update from revision scoring team that we
have sent to this mailing list.

New developments:

   - ORES review tool is deployed as a beta feature in Turkish Wikipedia.
   Now six Wikis have this tool. [2]


   - CopyPatrol tool soon will show ORES scores if they pass a certain
   threshold. [3]


   - We are talking about integration with Detox, feel free to chime in. [4]


Maintenance and robustness:

   - Currently we are dealing with increasing memory pressure on scb nodes.
   Actions we did to reduce this pressure:


   - We migrated most of RandomForrest models to GradientBoosting, which
   will reduce memory pressure greatly without affecting accuracy of models
   noticeably [5]


   - It seems there is a memory leak issue with celery. We bypassed that by
   setting a periodic restart of workers. [6]


   - We reduced maximum number of precaching requests in order to prevent
   spikes that might cause memory pressure on other services. [7]


   - We reduced number of web processes to 2/3. It is still fine. [8]


   - We finished up the refactor and it will soon goes to the production
   cluster. [9]


   - Damaging and goodfaith models had issue in Dutch Wikipedia. Got fixed.
   [10]


   - Our metrics collector now sends timed requests. We will have a
   dashboard in grafana for that soon. [11]


   - There will be a link to "ORES review tool" page [12] in legend of
   RecentChanges and Watchlist. [13]


   - When revscoring fails for any unknown reason, ORES return a proper
   message now. [14]


   - We fixed a puppet issue that caused trouble while creating new web
   instances. Got fixed [15]


1. It's 13. We are not superstitious, just kidding ;)
2. https://phabricator.wikimedia.org/T139992
3. https://phabricator.wikimedia.org/T139009
4. https://phabricator.wikimedia.org/T139007
5. https://phabricator.wikimedia.org/T139963
6. https://phabricator.wikimedia.org/T140020
7. https://gerrit.wikimedia.org/r/299559
8. https://gerrit.wikimedia.org/r/298739
9. https://github.com/wiki-ai/ores/pull/155
10. https://phabricator.wikimedia.org/T140038
11. https://phabricator.wikimedia.org/T137442
12. https://www.mediawiki.org/wiki/ORES_review_tool
13. https://phabricator.wikimedia.org/T140361
14. https://phabricator.wikimedia.org/T140301
15. https://phabricator.wikimedia.org/T140265

Sincerely,
Amir from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] The Revision Scoring weekly update

[mathieu stumpf guntz]

Hey, it looks like it works with the right URL ^^

Admittedly, I had manually changed the "en" to "fr" rather than using 
the "in other languages" tool box. /o\


Thank you Aaron.


Le 13/07/2016 à 17:15, Aaron Halfaker a écrit :

Looks like the frwiki version is at
https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Label, not
https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Labels :)

FWIW, the frwiki version is working for me just fine.  Let me know if it
still doesn't work at the appropriate URL.  Then, I think that Amir might
be right about conflicting gadgets.

On Wed, Jul 13, 2016 at 10:09 AM, Amir Ladsgroup 
wrote:


Hey,
You probably enabled a gadget that is broken causing wikilabels not working
properly (we had this issue in fawiki a while ago). I tested it and it was
okay in frwiki. First, try to disable your gadgets in frwiki and if that
was the case try to understand what is the broken gadget (e.g. check
console log). If that was not the case, please open a phabricator bug and
put console log. I will investigate it.

Best

On Wed, Jul 13, 2016 at 7:15 PM mathieu stumpf guntz <
psychosl...@culture-libre.org> wrote:


Hi Amir,

Could you give me more information on the wikilabels campaigns for
French WP? Is there a way to contribute on this? I did followed
instructions on documentation page
 to install the JS on meta
. Then when
I go on the English Wikipedia
, it works™. But on the
French version  it
doesn't.

What did I missed? ^^


Le 11/07/2016 à 21:40, Amir Ladsgroup a écrit :

Hey,
This is the 12th weekly update from revision scoring team that we have

sent

to this mailing list.

New developments:

 - We deployed ORES review tool in Russian and Portuguese Wikipedia!

[1]

 [2]


 - Basic support for English Wiktionary and Czech Wikipedia is live

in

 labs.[3]  Soon production will follow.


 - Damaging model for Polish Wikipedia is ready and soon be

available

 (then we can enable ORES review tool as well) [4]


 - ORES extension now highlights the whole row for Enhanced recent
 changes too. It will be live in two weeks [5]


 - We built prototype of a small web app to show progress of

wikilabels

 campaigns at

https://tools.wmflabs.org/dexbot/tools/wikilabels_stats.php.

 Feedback is welcome [6]


Maintenance and robustness:

 - If a web node goes down in labs, we get icigna warnings [7]


 - We re-launched wikilabels campaigns for French and Azeri

Wikipedia

[8]

 [9]


 - We are starting to use tsv2json to load the whole data when

adding

new

 campaigns in wikilabels [10]


 - "Hide/Show good edits" soon will be changed to "Hide/Show

probably

 good edits" in recent changes and watchlists to reflect ORES scores

more

 accurately [11]


1. https://phabricator.wikimedia.org/T139541
2. https://phabricator.wikimedia.org/T139692
3. https://phabricator.wikimedia.org/T139789
4. https://phabricator.wikimedia.org/T139207
5. https://phabricator.wikimedia.org/T139924
6. https://phabricator.wikimedia.org/T139874
7. https://phabricator.wikimedia.org/T134782
8. https://phabricator.wikimedia.org/T107730
9. https://phabricator.wikimedia.org/T139875
10. https://phabricator.wikimedia.org/T139939
11. https://phabricator.wikimedia.org/T139754

Sincerely,
Amir from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Looks like the frwiki version is at
https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Label, not
https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Labels :)

FWIW, the frwiki version is working for me just fine.  Let me know if it
still doesn't work at the appropriate URL.  Then, I think that Amir might
be right about conflicting gadgets.

On Wed, Jul 13, 2016 at 10:09 AM, Amir Ladsgroup 
wrote:

> Hey,
> You probably enabled a gadget that is broken causing wikilabels not working
> properly (we had this issue in fawiki a while ago). I tested it and it was
> okay in frwiki. First, try to disable your gadgets in frwiki and if that
> was the case try to understand what is the broken gadget (e.g. check
> console log). If that was not the case, please open a phabricator bug and
> put console log. I will investigate it.
>
> Best
>
> On Wed, Jul 13, 2016 at 7:15 PM mathieu stumpf guntz <
> psychosl...@culture-libre.org> wrote:
>
> > Hi Amir,
> >
> > Could you give me more information on the wikilabels campaigns for
> > French WP? Is there a way to contribute on this? I did followed
> > instructions on documentation page
> >  to install the JS on meta
> > . Then when
> > I go on the English Wikipedia
> > , it works™. But on the
> > French version  it
> > doesn't.
> >
> > What did I missed? ^^
> >
> >
> > Le 11/07/2016 à 21:40, Amir Ladsgroup a écrit :
> > > Hey,
> > > This is the 12th weekly update from revision scoring team that we have
> > sent
> > > to this mailing list.
> > >
> > > New developments:
> > >
> > > - We deployed ORES review tool in Russian and Portuguese Wikipedia!
> > [1]
> > > [2]
> > >
> > >
> > > - Basic support for English Wiktionary and Czech Wikipedia is live
> in
> > > labs.[3]  Soon production will follow.
> > >
> > >
> > > - Damaging model for Polish Wikipedia is ready and soon be
> available
> > > (then we can enable ORES review tool as well) [4]
> > >
> > >
> > > - ORES extension now highlights the whole row for Enhanced recent
> > > changes too. It will be live in two weeks [5]
> > >
> > >
> > > - We built prototype of a small web app to show progress of
> > wikilabels
> > > campaigns at
> > https://tools.wmflabs.org/dexbot/tools/wikilabels_stats.php.
> > > Feedback is welcome [6]
> > >
> > >
> > > Maintenance and robustness:
> > >
> > > - If a web node goes down in labs, we get icigna warnings [7]
> > >
> > >
> > > - We re-launched wikilabels campaigns for French and Azeri
> Wikipedia
> > [8]
> > > [9]
> > >
> > >
> > > - We are starting to use tsv2json to load the whole data when
> adding
> > new
> > > campaigns in wikilabels [10]
> > >
> > >
> > > - "Hide/Show good edits" soon will be changed to "Hide/Show
> probably
> > > good edits" in recent changes and watchlists to reflect ORES scores
> > more
> > > accurately [11]
> > >
> > >
> > > 1. https://phabricator.wikimedia.org/T139541
> > > 2. https://phabricator.wikimedia.org/T139692
> > > 3. https://phabricator.wikimedia.org/T139789
> > > 4. https://phabricator.wikimedia.org/T139207
> > > 5. https://phabricator.wikimedia.org/T139924
> > > 6. https://phabricator.wikimedia.org/T139874
> > > 7. https://phabricator.wikimedia.org/T134782
> > > 8. https://phabricator.wikimedia.org/T107730
> > > 9. https://phabricator.wikimedia.org/T139875
> > > 10. https://phabricator.wikimedia.org/T139939
> > > 11. https://phabricator.wikimedia.org/T139754
> > >
> > > Sincerely,
> > > Amir from the Revision Scoring team
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] The Revision Scoring weekly update

[Amir Ladsgroup]

Hey,
You probably enabled a gadget that is broken causing wikilabels not working
properly (we had this issue in fawiki a while ago). I tested it and it was
okay in frwiki. First, try to disable your gadgets in frwiki and if that
was the case try to understand what is the broken gadget (e.g. check
console log). If that was not the case, please open a phabricator bug and
put console log. I will investigate it.

Best

On Wed, Jul 13, 2016 at 7:15 PM mathieu stumpf guntz <
psychosl...@culture-libre.org> wrote:

> Hi Amir,
>
> Could you give me more information on the wikilabels campaigns for
> French WP? Is there a way to contribute on this? I did followed
> instructions on documentation page
>  to install the JS on meta
> . Then when
> I go on the English Wikipedia
> , it works™. But on the
> French version  it
> doesn't.
>
> What did I missed? ^^
>
>
> Le 11/07/2016 à 21:40, Amir Ladsgroup a écrit :
> > Hey,
> > This is the 12th weekly update from revision scoring team that we have
> sent
> > to this mailing list.
> >
> > New developments:
> >
> > - We deployed ORES review tool in Russian and Portuguese Wikipedia!
> [1]
> > [2]
> >
> >
> > - Basic support for English Wiktionary and Czech Wikipedia is live in
> > labs.[3]  Soon production will follow.
> >
> >
> > - Damaging model for Polish Wikipedia is ready and soon be available
> > (then we can enable ORES review tool as well) [4]
> >
> >
> > - ORES extension now highlights the whole row for Enhanced recent
> > changes too. It will be live in two weeks [5]
> >
> >
> > - We built prototype of a small web app to show progress of
> wikilabels
> > campaigns at
> https://tools.wmflabs.org/dexbot/tools/wikilabels_stats.php.
> > Feedback is welcome [6]
> >
> >
> > Maintenance and robustness:
> >
> > - If a web node goes down in labs, we get icigna warnings [7]
> >
> >
> > - We re-launched wikilabels campaigns for French and Azeri Wikipedia
> [8]
> > [9]
> >
> >
> > - We are starting to use tsv2json to load the whole data when adding
> new
> > campaigns in wikilabels [10]
> >
> >
> > - "Hide/Show good edits" soon will be changed to "Hide/Show probably
> > good edits" in recent changes and watchlists to reflect ORES scores
> more
> > accurately [11]
> >
> >
> > 1. https://phabricator.wikimedia.org/T139541
> > 2. https://phabricator.wikimedia.org/T139692
> > 3. https://phabricator.wikimedia.org/T139789
> > 4. https://phabricator.wikimedia.org/T139207
> > 5. https://phabricator.wikimedia.org/T139924
> > 6. https://phabricator.wikimedia.org/T139874
> > 7. https://phabricator.wikimedia.org/T134782
> > 8. https://phabricator.wikimedia.org/T107730
> > 9. https://phabricator.wikimedia.org/T139875
> > 10. https://phabricator.wikimedia.org/T139939
> > 11. https://phabricator.wikimedia.org/T139754
> >
> > Sincerely,
> > Amir from the Revision Scoring team
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] The Revision Scoring weekly update

[mathieu stumpf guntz]

Hi Amir,

Could you give me more information on the wikilabels campaigns for 
French WP? Is there a way to contribute on this? I did followed 
instructions on documentation page 
 to install the JS on meta 
. Then when 
I go on the English Wikipedia 
, it works™. But on the 
French version  it 
doesn't.


What did I missed? ^^


Le 11/07/2016 à 21:40, Amir Ladsgroup a écrit :

Hey,
This is the 12th weekly update from revision scoring team that we have sent
to this mailing list.

New developments:

- We deployed ORES review tool in Russian and Portuguese Wikipedia! [1]
[2]


- Basic support for English Wiktionary and Czech Wikipedia is live in
labs.[3]  Soon production will follow.


- Damaging model for Polish Wikipedia is ready and soon be available
(then we can enable ORES review tool as well) [4]


- ORES extension now highlights the whole row for Enhanced recent
changes too. It will be live in two weeks [5]


- We built prototype of a small web app to show progress of wikilabels
campaigns at https://tools.wmflabs.org/dexbot/tools/wikilabels_stats.php.
Feedback is welcome [6]


Maintenance and robustness:

- If a web node goes down in labs, we get icigna warnings [7]


- We re-launched wikilabels campaigns for French and Azeri Wikipedia [8]
[9]


- We are starting to use tsv2json to load the whole data when adding new
campaigns in wikilabels [10]


- "Hide/Show good edits" soon will be changed to "Hide/Show probably
good edits" in recent changes and watchlists to reflect ORES scores more
accurately [11]


1. https://phabricator.wikimedia.org/T139541
2. https://phabricator.wikimedia.org/T139692
3. https://phabricator.wikimedia.org/T139789
4. https://phabricator.wikimedia.org/T139207
5. https://phabricator.wikimedia.org/T139924
6. https://phabricator.wikimedia.org/T139874
7. https://phabricator.wikimedia.org/T134782
8. https://phabricator.wikimedia.org/T107730
9. https://phabricator.wikimedia.org/T139875
10. https://phabricator.wikimedia.org/T139939
11. https://phabricator.wikimedia.org/T139754

Sincerely,
Amir from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Amir Ladsgroup]

Hey,
This is the 12th weekly update from revision scoring team that we have sent
to this mailing list.

New developments:

   - We deployed ORES review tool in Russian and Portuguese Wikipedia! [1]
   [2]


   - Basic support for English Wiktionary and Czech Wikipedia is live in
   labs.[3]  Soon production will follow.


   - Damaging model for Polish Wikipedia is ready and soon be available
   (then we can enable ORES review tool as well) [4]


   - ORES extension now highlights the whole row for Enhanced recent
   changes too. It will be live in two weeks [5]


   - We built prototype of a small web app to show progress of wikilabels
   campaigns at https://tools.wmflabs.org/dexbot/tools/wikilabels_stats.php.
   Feedback is welcome [6]


Maintenance and robustness:

   - If a web node goes down in labs, we get icigna warnings [7]


   - We re-launched wikilabels campaigns for French and Azeri Wikipedia [8]
   [9]


   - We are starting to use tsv2json to load the whole data when adding new
   campaigns in wikilabels [10]


   - "Hide/Show good edits" soon will be changed to "Hide/Show probably
   good edits" in recent changes and watchlists to reflect ORES scores more
   accurately [11]


1. https://phabricator.wikimedia.org/T139541
2. https://phabricator.wikimedia.org/T139692
3. https://phabricator.wikimedia.org/T139789
4. https://phabricator.wikimedia.org/T139207
5. https://phabricator.wikimedia.org/T139924
6. https://phabricator.wikimedia.org/T139874
7. https://phabricator.wikimedia.org/T134782
8. https://phabricator.wikimedia.org/T107730
9. https://phabricator.wikimedia.org/T139875
10. https://phabricator.wikimedia.org/T139939
11. https://phabricator.wikimedia.org/T139754

Sincerely,
Amir from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Amir Ladsgroup]

Hey,
This is the 11th weekly update from revision scoring team that we have sent
to this mailing list.

*New developments:*

   - ORES review tool as a beta feature is enabled in Dutch Wikipedia. More
   wikis to come soon this week [1].


   - We have basic edit quality model for Czech Wikipedia ready and merged.
   To be deployed this week [2].


   - We also have basic models for English Wiktionary too. This is the
   second non-Wikipedia project we support after Wikidata [3].


   - Thanks to Tar Lócesilion, we have Polish edit quality campaign
   completed, We are working on building damaging and goodfaith models at the
   moment [4].


*Maintenance and robustness:*

   - We decreased our web capacity in order to reduce memory pressure on
   scb nodes. You should not get any overload error since our capacity is
   still very high but if you do, please contact us immediately and we will
   bring it back up [5].


   - We improved documentation on ores.wikimedia.org page a little bit. To
   be deployed this week [6].


We are working on a rather big refactor on ores which will give us
performance boost on scoring multiple models at the same time [7] and
reduce memory usage [8]. Feel free to chime in and give us feedback [9].

1. https://phabricator.wikimedia.org/T139432
2. https://phabricator.wikimedia.org/T138885
3. https://phabricator.wikimedia.org/T138630
4. https://phabricator.wikimedia.org/T130269
5. https://phabricator.wikimedia.org/T139177
6. https://phabricator.wikimedia.org/T138089
7. https://phabricator.wikimedia.org/T134606
8. https://phabricator.wikimedia.org/T139407
9. https://phabricator.wikimedia.org/T139408

Sincerely,
Amir from the Revision Scoring team.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The revision scoring weekly update

[Amir Ladsgroup]

Hey all,
This is the 10th weekly update (yay!) from revision scoring team that we
have sent to this mailing list.

We had Wikimania so it was a busy week.

Sessions:

   - Lessons Learned building machine models for Wikidata [1] Video [2]
   Slides [3]


   - Hackathon: Supporting new languages in ORES [4] Video [5]



New developments:

   - Thanks to the services team, we now use ChangePropagation for
   precaching [6]


   - New version of ORES is deployed in so new edit quality models (svwiki
   and nowiki) and article quality models (ruwiki) and other new deployments
   are now live. Labs: [7] Prod [8]


   - ores in beta cluster is live again with the exact same settings as
   prod so we can test more carefully [9]


   - We launched edit quality campaign for Swedish Wikipedia [10]


   - Thanks to Daniel Zahn, we have monitoring for worker nodes in
   production cluster [11]


   - Our precached daemon in labs now support omitting bot edits [12]


   - Revscoring supports Czech language, allow us enabling basic support
   for cswiki soon. [13]


Maintenance and robustness:

   - We shipped all of GUI dependencies including Swagger, Wikimedia UI and
   bootstrap to ORES itself [14]


   - Fixed Swagger specs for better monitoring by Ops [15]


   - Fixed puppet issue with worker nodes in labs [16]


   - Wikilabels campaign in huwiki was broken. Got fixed [17]


   - URLs without trailing slash won't get redirected and just return the
   same response [18]


   - ores-wikimedia-config repo got renamed to ores-wmflabs-config to show
   the distinction better [19]


   - We improved some documentation with regard to deployment, in order to
   prevent down times [20]



1. https://phabricator.wikimedia.org/T138351
2. https://www.youtube.com/watch?v=dPs1evXFZa0
3.
https://commons.wikimedia.org/wiki/File:Lessons_learned_building_machine_learning_models_for_Wikidata.pdf
4. https://phabricator.wikimedia.org/T134628
5. https://www.youtube.com/watch?v=WnM5Ys2-Cb8
6. https://phabricator.wikimedia.org/T138090
7. https://phabricator.wikimedia.org/T137828
8. https://phabricator.wikimedia.org/T138863
9. https://phabricator.wikimedia.org/T138445
10. https://phabricator.wikimedia.org/T131451
11. https://phabricator.wikimedia.org/T138882
12. https://phabricator.wikimedia.org/T137967
13. https://phabricator.wikimedia.org/T131637
14. https://phabricator.wikimedia.org/T137362
15. https://phabricator.wikimedia.org/T137804
16. https://phabricator.wikimedia.org/T137813
17. https://phabricator.wikimedia.org/T138576
18. https://phabricator.wikimedia.org/T138682
19. https://phabricator.wikimedia.org/T138253
20. https://phabricator.wikimedia.org/T138234

Sinceraly
Amir from the Revision Scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The revision scoring weekly update

[Aaron Halfaker]

Hey folks,

This is the 9th weekly update for the revision scoring team that we have
sent to this mailing list.

The biggest news is that we've officially deployed the ORES review tool[1]
to Persian Wikipedia and Wikidata[2,9].  We'll be posting a more complete
announcement to Wikitech-l soon.

New developments:

   - We generalized how the wsgi and celery applications are configured[4]


   - We fixed the way Special:Contributions is rendered so that the ORES
   review tool can work better there[6]


Maintenance and robustness:

   - We documented some downtime for the ORES production service[3]


   - We untangled and simplified our deployment repositories[5]


   - An issue with Wikilabels accepting too many labels for a single task
   is fixed[7]


   - We brought our deployment of ORES in the beta cluster closer to
   production[8]


   - We solved an issue with how the ORES extension populates scores[10]


Amir will be hosting a workshop for extending ORES support at Wikimania.
See https://phabricator.wikimedia.org/T134628 for more details.

1. https://www.mediawiki.org/wiki/ORES_review_tool
2. https://phabricator.wikimedia.org/T130210
3. https://phabricator.wikimedia.org/T138248
4. https://phabricator.wikimedia.org/T137820
5. https://phabricator.wikimedia.org/T137819
6. https://phabricator.wikimedia.org/T122537
7. https://phabricator.wikimedia.org/T131997
8. https://phabricator.wikimedia.org/T137821
9. https://phabricator.wikimedia.org/T130212
10. https://phabricator.wikimedia.org/T137854

-Aaron
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The revision scoring weekly update

[Amir Ladsgroup]

Hi folks,

This is the 8th weekly update for the revision scoring team that we have
sent to this mailing list.

*New developments:*

   - ORES extension got deployed in Persian Wikipedia. [1] Give it a try!
   [2]


   - Article quality model ("wp10" model) is now working for Russian
   Wikipedia. [3] It will be deployed this week


   - We deplyed article topic campaign for English Wikipedia [4]


   - ores.wikimedia.org does have a grafana dashboard now [5]


*Maintenance and robustness:*

   - ORES icinga didn't work when workers were down[13], it got fixed [6]


   - We finished load testing ores.wikimedia.org and it was quite alright
   [7] [8]


   - CORS is moved to uwsgi level to it works in prod too [9]


   - Deploying new versions of ORES in prod and labs has a proper
   documentation page now [10] [11]


   - We had intermittent spikes of errored revisions, got it resolved [12]


1. https://phabricator.wikimedia.org/T130211
2.
https://fa.wikipedia.org/wiki/Special:Preferences#mw-prefsection-betafeatures
3. https://phabricator.wikimedia.org/T131635
4. https://phabricator.wikimedia.org/T137325
5. https://phabricator.wikimedia.org/T137367
6. https://phabricator.wikimedia.org/T137592
7. https://phabricator.wikimedia.org/T137365
8. https://phabricator.wikimedia.org/T137131
9. https://phabricator.wikimedia.org/T137433
10. https://phabricator.wikimedia.org/T137570
11. https://wikitech.wikimedia.org/wiki/Ores/Deployment
12. https://phabricator.wikimedia.org/T134109
13. https://wikitech.wikimedia.org/wiki/Incident_documentation/20160610-ORES

Best,
Amir
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision Scoring weekly update

[Aaron Halfaker]

Hi folks,

This is the 6th weekly update for the revision scoring team that we have
sent to this mailing list.

*New developments:*

   - We set up a system for expiring labeling tasks after they have not
   been completed [3]


   - We set up the ORES service in the Beta cluster [8]


*Maintenance and robustness:*

   - We brought ores-beta (in the beta cluster) back online [1]


   - We solved a problem with ORES being able to score multiple revisions
   in batch [2]


   - We addressed some database connection issues in Wiki labels [4]


   - We brought Wikilabels back online after some deployment issues [5] and
   wrote up a post-mortem [6]


   - We addressed some puppet issues with ORES' staging environment [7]



   1. https://phabricator.wikimedia.org/T135677
   2. https://phabricator.wikimedia.org/T136278
   3. https://phabricator.wikimedia.org/T135400
   4. https://phabricator.wikimedia.org/T136502
   5. https://phabricator.wikimedia.org/T136520
   6. https://phabricator.wikimedia.org/T136523
   7. https://phabricator.wikimedia.org/T136488
   8. https://phabricator.wikimedia.org/T130404


Stay tuned!
-Aaron
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Revision scoring weekly update

[Amir Ladsgroup]

Hey,
This is the fifth weekly update for revision scoring team that we sent to
this mailing list.

*New developments:*

   - We got Swedish basic model ready to deloy, likely to happen in the
   next week [1] [2]


   - We generated list of bad words for every Wikipedia with more than 100K
   articles (with a few exceptions) [3]


*Maintenance and robustness:*

   - We enabled CORS for Wikimedia wikis in wikilabels and now we won't let
   you do write actions via GET [4]


   - We are using systemd watchdogs in precaching to be sure it stays
   alive. [5]


   - We are changing some settings in nginx and uwsgi in order to finalize
   moving to prod [6]


1. https://phabricator.wikimedia.org/T131450
2. https://phabricator.wikimedia.org/T135604
3. https://phabricator.wikimedia.org/T134629
4. https://phabricator.wikimedia.org/T135377
5. https://phabricator.wikimedia.org/T135941
6. https://phabricator.wikimedia.org/T135655


Sincerely,
The Revision scoring team
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l