Dear Wikidata Community, We are trying to understand which database technologies and strategies Wikidata uses for storing, updating, and querying the data (knowledge) it manipulates.
By looking at the documentation<https://wmde.github.io/wikidata-wikibase-architecture/assets/img/03-dataflow-out.drawio.17c12ee9.svg> we understood that RDF is only used for the Wikidata Query Service, but we could not find out exactly how Wikidata stores the information that is translated to RDF during the data dump. More specifically, we understood that a MySQL (or is it MariaDB?) relational database is used as the key persistence component for most of Wikidata services and that the information that is maintained in this database is periodically exported to multiple formats, including RDF. In addition, looking at the relational database schema published in the documentation<https://www.mediawiki.org/wiki/Manual:Database_layout> we could not locate tables that are easily mappable to the Wikibase Data Model<https://www.mediawiki.org/wiki/Wikibase/DataModel>. Thus, we hypothesize that there is some software component (Wikibase Common Data Access?) that dynamically translates the data contained in those tables to Statements, Entities, etc. Is that hypothesis, correct? If yes, does this software component uses any intermediate storage mechanism for caching those Statements, Entities, ...? Or are those translations always performed at runtime on-the-fly (be it for querying, adding, or updating Statements, Entities, …)? Finally, we would like to understand more about how Wikidata REST API<https://www.wikidata.org/wiki/Wikidata:REST_API> is implemented: * In which database are the statements that added/retrieved through it stored? Is it being stored in the central MySQL database or in another database? * Does it have any support for pagination of statements? For example, if an item has many statements associated with a property, does the API assumes that both the underlying database and the network will support the retrieval of all those statements? * Are you currently considering implementing the support for more flexible querying of statements, or such requirement has been fully delegated to the Wikidata Query Service? If there is an updated documentation that could help us answer those questions, could you kindly point us to it? Otherwise, would you be able to share this information with us? Best Regards, Elton F. de S. Soares Advisory Software Engineer Rio de Janeiro, RJ, Brazil IBM Research E-mail: elt...@ibm.com<mailto:elt...@ibm.com>
_______________________________________________ Wikidata-tech mailing list -- wikidata-tech@lists.wikimedia.org To unsubscribe send an email to wikidata-tech-le...@lists.wikimedia.org
