Ladsgroup added a subscriber: guergana.tzatchkova.
Ladsgroup added a comment.
So I have been working on this (plus pairing with @guergana.tzatchkova) .
What we did is that first we added a new model in the make file:
datasets/wikidatawiki.labelings.5k.2020.json:
./utility fetch_labels \
https://labels.wmflabs.org/campaigns/wikidatawiki/95/ > $@
datasets/wikidatawiki.labeling_revisions.w_cache.5k.2020.json: \
datasets/wikidatawiki.labelings.5k.2020.json
cat $< | \
revscoring extract \
articlequality.feature_lists.wikidatawiki.item_quality \
--host https://www.wikidata.org \
--batch-size 10 \
--verbose > $@
models/wikidatawiki.item_quality.gradient_boosting.2020.model: \
datasets/wikidatawiki.labeling_revisions.w_cache.5k.2020.json
cat $< | \
revscoring cv_train \
revscoring.scoring.models.GradientBoosting \
articlequality.feature_lists.wikidatawiki.item_quality \
item_quality \
--version $(item_quality_major_minor).0 \
-p 'learning_rate=0.01' \
-p 'n_estimators=500' \
-p 'max_features="log2"' \
-p 'max_depth=5' \
--labels '"A","B","C","D","E"' \
--center --scale > $@
revscoring model_info $@ > model_info/wikidatawiki.item_quality.2020.md
Which built the model based on the 1300 observations. Then we ran this code:
import mwapi
from revscoring import Model
import time
import json
from revscoring.extractors.api.extractor import Extractor
import requests
r =
requests.get('https://labels.wmflabs.org/campaigns/wikidatawiki/95/?tasks=')
val = r.json()
with open("models/wikidatawiki.item_quality.gradient_boosting.2020.model")
as f:
scorer_model = Model.load(f)
extractor = Extractor(mwapi.Session(host="https://www.wikidata.org";,
user_agent="revscoring demo"))
labeleds = []
for i in val['tasks']:
if i['labels']:
continue
try:
feature_values = list(extractor.extract(i['data']['rev_id'],
scorer_model.features))
feature_values = scorer_model.score(feature_values)
except:
continue
if feature_values['probability'][feature_values['prediction']] < 0.9:
continue
i['labels'] = [
{
'data': {'item_quality': feature_values['prediction']},
'timestamp': time.time(),
'autolabel': True
}
]
labeleds.append(i)
with open('labeleds.json', 'w') as f:
f.write(json.dumps(labeleds))
With this it labeled around 1237 new items. This really quick python script
produced the postgres command we needed to run on production to add the labels:
import json
with open('labeleds.json', 'r') as f:
cases = json.loads(f.read())
print('insert into label (task_id, user_id, timestamp, data) values')
for case in cases:
print('(' + str(case['id']) + ', 13387278, NOW(), \''+
json.dumps(case['labels'][0]['data']) + '\'), ')
print(';')
(The user id is ID of my bot, zero causes error sometimes and I needed
something distinct to be able to clean up later)
Then I'm running this all over again (the second iteration).
TASK DETAIL
https://phabricator.wikimedia.org/T261293
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup
Cc: guergana.tzatchkova, Aklapper, Lydia_Pintscher, Hazizibinmahdi, Akuckartz,
darthmon_wmde, Michael, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden,
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Ladsgroup,
Mbch331
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
