| Jheald added a comment. |
The attached subtickets are an interesting read. They all seem to be based on taking the Q-number value of "depicts", storing it as a string in the text-search index, and then doing an indexed string-match for it. Of course first baby-steps are important, and this facility will be crucial to be able to confirm correct entry, storage, and direct retrievability of "depicts" values.
But it seems a long way short of the functionality that has generally been assumed for retrieval based on "depicts" values, and that is ultimately going to be needed.
Has the team had any initial thoughts what possible strategies are likely to be available or preferred or required, to make such more general retrieval achievable, and what sort of back-end requirements may be needed to make the system acceptably responsive (ie near-instant returns), and able to cope with full production load? Are there tickets open for these questions anywhere?
To give an idea of the sort of issue that's motivating my question, consider a user search for "man with hat".
One of the images we might hope to see included in the set returned to the user might be File:Giovanni Bellini, portrait of Doge Leonardo Loredan.jpg.
But the CommonsData for this file will most probably not include the literal statements "depicts man" or "depicts hat".
Instead, most probably, it would have the statement "depicts Q1759759", the Wikidata item for the painting.
Even if it was described in-situ, it would most probably have the statement "depicts Q250210" (Leonardo Loredan) rather than "depicts man"; and depicts (or P1354 "shown with features") Q1134210 "doge's cap".
A simple string search for depicts Q8441 "man" or depicts Q80151 "hat" is not going to match it.
It would seem that, at the very least, search is going to need to match to any items in the wikidata subclass (P279) trees of the search terms.
(As an aside, it may be worth remembering that Blazegraph can get into difficulties (T116298) if queries combine multiple path expansions (as all of ours would), without careful hinting to explore such expansions in a many-to-few direction. It also (for some reason) sometimes finds it much quicker to traverse such paths if they are given in reverse (ie ?b ^prop* ?a rather than ?a prop* ?b), even when a direction hint is given. So this might need care.)
A further complication is that male individuals on wikidata are not represented as instances of "man" (Q8441), but rather as instances of "human" (Q5), with property "sex or gender" (P21) = "male" (Q6581097).
The user interface will therefore presumably need to lead the user away from searching for "man with hat" towards "human being who is male with hat". Similarly, if a user is searching for images that depict "bald politician", wikidata does not represent individuals as instances (P31) of "politician" (Q82955); instead they are instances of "human" (Q5), with "occupation" (P106) = "politician". The faceted search UI is going to need to make it a lot easier for people to enter "human" and then "occupation", rather than "politician".
But the big question, at least for me: are the team confident that searches like these are going to be deliverable in reasonable time; and still deliverable in reasonable time at production load?
Is it a concern, for example, that anything that requires materialising the whole set of instances of Q5 "human" is inevitably going to be slow -- the number of such items on Wikidata is currently 4.375 million, and Commons will add even more. Thus, for example, just to count the number of such items with preferred images (P18) currently takes no less than 50 seconds (tinyurl.com/y7ddk8wm).
Of course, even with searches involving instances of Q5, it might not be necessary to materialise the full set, if the final result-set will ultimately be LIMITed to only, say, 2000 results. (Though beware that currently the labels SERVICE requires the entire results set to be materialised, even with a LIMIT directive). Also it's true that other clauses in the search (if present) may well be more restrictive, and therefore (if identified by the optimiser) may open up a faster solution path. Nevertheless, some searches might still require quite deep tree explorations, materialisations of quite big intermediate sets, and quite big merges, even to produce only a few hundred results.
As a team, do we have a target for how quickly results need to start being returned from such a search, if the system is to be considered decently responsive?
Are we confident that we think this should be achievable, for the faceted search back-end running in the full context of the entire wikidata dataset plus a maximal level of image description in CommonsData?
And how many such searches do we anticipate the system will need to be able to field an hour, in full production use?
(A first estimate might perhaps be some multiple of the number of Commons category pages currently served an hour; though if faceted search becomes as friendly and intuitive and powerful a tool as people hope, the hope must be that it would soon become considerably more popular than this).
Cc: Jheald, Ramsey-WMF, Cparle, Aklapper, Lahi, PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, V4switch, LawExplorer, Susannaanas, Wong128hk, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Matanya, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
