Hi James, hi all, the relationship and synergies of SMW and Wikidata are of course quite important to us, so I am taking the time to write a slightly longer email that I also send to the user list.
At the moment, Wikidata is still under development, so it is a bit early to make concrete plans on how to align the project. Here are some things that I can already answer now: (1) What is the software in "Wikidata"? The software underlying Wikidata is called Wikibase. It consists of two extensions: Repository (where data is stored; this will run at wikidata.org) and Client (that pulls data into other MediaWiki sites; this will run at wikipedia.org). Both extensions will be installable on MediaWiki sites to get Wikidata-like features there. (2) What are the main differences between SMW and Wikidata? Wikidata is being built with SMW in mind, and the experiences with SMW are a valuable input for its design. The main use case of Wikidata (a centralised, multi-lingual site that serves as a data repository) is different from that of SMW (a data-enhanced MediaWiki), and this leads to a number of differences. The data model of Wikidata is a bit more complex than that of SMW. For example, every fact will also have an (optional) reference. Wikidata will also use more complex datatypes (e.g., since many text values can be translated). However, both Wikidata and SMW are still mostly based on property-value assignments, and Wikidata will have a Property namespace. Also, the more complex structures in Wikidata could be captured in SMW using internal objects. The user interface of Wikibase Repository will be based on input forms, and thus quite different from SMW. The UI will be a bit like a full-page, editable Factbox, but much more advanced, with a lot of support for entering values. It is not defined yet what kind of query language Wikidata will support in Phase 3 (or thereafter). This needs to be designed first, but the more complicated data model will also have an impact there. (3) What is a ContentHandler and how could it help SMW? ContentHandlers allow MediaWiki to manage textual content that is not interpreted as wikitext. Wikidata uses this to store data in a text-based format (JSON) without having to invent a MediaWiki syntax extension. In the future, it might be possible to have "multi-part" content for one page, so that one can store both Wikitext and other structured data in a cleaner way. SMW could use this to allow data to be stored directly for a page, without having to specify the data in the wikitext. This could be a bit like an "editable Factbox". MediaWiki could do something similar with Categories. (4) Will ContentHandlers reduce the dependence on custom SQL? No. Content that is stored in textual form on wiki pages is not suitable for answering queries quickly. To do this, it will always be necessary to have custom tables and/or external storage engines that provide fast access to the data. This is also the case for Wikidata. Currently, the MW page is the *primary* storage for SMW data, while the SQL tables are just *secondary* storage for faster lookup. The ContentHandlers can improve primary storage handling, but won't affect secondary storages and querying. (5) What are the possible synergies and joint uses of Wikidata and SMW? The details can only be worked out when Wikidata is a bit more stable than it is now, but there are a few obvious scenarios: * Wikibase Repository + SMW: data given in SMW annotations and data edited in Wikidata end up in the same secondary data structures, and can be retrieved through the same interfaces/queries. There are three main options: SMW tables are used for Wikibase content (this would yield query support via #ask), Wikibase tables are used for SMW content (yields full integration with Wikibase data access features), or both. * SMW as a Wikibase Repository: SMW could support the use of Wikibase APIs and exports on top of its native data store, so that other sites could fetch SMW data via the Wikidata Client. * SMW as a Wikibase Client: SMW could also pull data from Wikibase Repositories to use it locally and to display it in its interfaces. * Exchanging code: Besides application-level integration, SMW and Wikidata are also going to exchange lines of code. Initially, Wikidata will exploit code from SMW (e.g., the calendar conversion code for times and dates). Later, smaller pieces of Wikidata code might be used for SMW. In many cases, this will mean that the code is adapted heavily; if it can be used without much change, one would possibly create a "library extension" (like Validator). (6) Will Wikidata replace SMW in the long run? No. If anything, the integration between both could become so good that the distinction is no longer relevant and we are only talking about different data authoring and data access interface extensions to MediaWiki, which users can enable and disable at their leisure. This will still take some time though. (7) What is the development plan for SMW with respect to Wikidata? The strategy is to wait until Wikidata has settled a bit before approaching any integration. This will probably be another 10 or 11 months from now (note that data management only comes in at Phase 2 of Wikidata, i.e., in 5 months). Exchanging some code might be possible before that, but integrating Wikibase while it is a moving target would not be feasible (there would not be a chance of keeping up with the speed of Wikidata, esp. since half the SMW team is engaged in Wikidata development right now). I hope this answers most questions on that topic for now. Maybe it would be nice to have a wikipage on semantic-mediawiki.org with these answers. Best regards, Markus On 30/04/12 23:44, James HK wrote: > Hi, > > ==WD ContentHandler == > Now that the Wikidata project is introducing a new ContentHandler [1], > we where curious if in near future SMW would make use of those methods > to store its own objects in the same way and reduces its reliance on > non-mediawiki tables/methods. > > As Wikidata and SMW share the same bottlenecks in terms of > performance, caching, scalability it might prove beneficial for SMW to > use similar methods to store and fetch dataobjects from a mw-database > while at the same time reduces its reliance on SQL specific code and > lift the ban on other supported database systems(Oracle,Sqlite etc.). > > The reason why we are asking is that we hope that some of the work > that has be done in SMW is reflected in Wikidata as visa versa to > ensure that both developments share some similarities to the extend > that their "plug and playable" (meaning causing the least > interruptions when installed together and make it easier for code > reviewers to understand the inner workings which increases stability). > > ==SMW and WD == > Furthermore, with the Wikidata project on the horizon, we would be > interested to see how SMW positioned and distinguished itself from WD > in light of recent developments. (A simple table could help to explain > those differences fairly quick while at the same time allows some > directional guidance for the future of SMW.) > > ==Validator== > The Validator provides validation methods for datatypes (and parameter > definitions etc.) and WD certainly needs similar methods, is it > planned that those methods (or part of it) are being moved into the > core? > > [1] > https://gerrit.wikimedia.org/r/gitweb?p=mediawiki%2Fcore.git;a=commit;h=337f65c03b59eaddc038bc1a0fb150dc97e03154 > > Cheers, > > mwjames > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Semediawiki-devel mailing list > Semediawiki-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/semediawiki-devel > ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
