Hello dear Wikidata enthusiasts,
I'm one of the authors of a recent paper where we made heavy use of
Wikidata to create a large-scale image-text dataset for training CLIP
vision-language models. I thought this list might find the approach
interesting.
What we did: We used SPARQL queries over the subclass-of (P279) and
parent-taxon (P171) hierarchies to extract ~135k visual entities from 21
super-entity categories (animals, plants, tools, buildings, etc.). For
each entity we collected its name, description, aliases, and sitelink
count. We then extracted attributes from Wikidata related to color,
partonomy, behavior, and other aspects. These entities and attributes
were used to generate search queries for downloading images from the web.
The resulting EntityNet dataset contains 33M images, 46M text
descriptions, and 613k text labels linked back to Wikidata entities. The
dataset and trained models are openly available on HuggingFace:
https://huggingface.co/datasets/lmb-freiburg/entitynet
<https://huggingface.co/datasets/lmb-freiburg/entitynet>
A key finding was that training on a mix of Wikidata's structured entity
information and noisy web alt-texts works better than either alone. The
knowledge graph metadata genuinely improves model quality.
Paper: https://arxiv.org/abs/2505.02746
<https://arxiv.org/abs/2505.02746> | Code:
https://github.com/lmb-freiburg/entitynet
<https://github.com/lmb-freiburg/entitynet>
Happy to answer any questions about how we used Wikidata or discuss
potential improvements to the entity extraction.
_______________________________________________
Wikidata mailing list -- [email protected]
Public archives at
https://lists.wikimedia.org/hyperkitty/list/[email protected]/message/3CMRFILT6MYLTVTKJKHZEQJVJQTATQNJ/
To unsubscribe send an email to [email protected]
