[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 383364 merged by jenkins-bot: [mediawiki/extensions/WikibaseLexeme@master] Bind against FieldDefinitions interface instead of implementation https://gerrit.wikimedia.org/r/383364TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 384047 merged by jenkins-bot: [mediawiki/extensions/WikibaseMediaInfo@master] Bind against FieldDefinitions interface instead of implementation https://gerrit.wikimedia.org/r/384047TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. This is merged and the config is enabled, but not reindexed yet, probably will take several days until it's done, the wikidata index is huge.TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-10-16T18:18:34Z] Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:383464|Add configuration for statement indexing for Wikidata]] T175199 (duration: 00m 47s)TASK

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 383464 merged by jenkins-bot: [operations/mediawiki-config@master] Add configuration for statement indexing for Wikidata https://gerrit.wikimedia.org/r/383464TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 384516 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Make Item… and PropertyFieldDefinitions accept arrays https://gerrit.wikimedia.org/r/384516TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 382725 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Optimize StatementsField for performance and readability https://gerrit.wikimedia.org/r/382725TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 384516 had a related patch set uploaded (by Thiemo Mättig (WMDE); owner: Thiemo Mättig (WMDE)): [mediawiki/extensions/Wikibase@master] Make Item… and PropertyFieldDefinitions accept arrays https://gerrit.wikimedia.org/r/384516TASK

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 384047 had a related patch set uploaded (by Thiemo Mättig (WMDE); owner: Thiemo Mättig (WMDE)): [mediawiki/extensions/WikibaseMediaInfo@master] Bind against FieldDefinitions interface instead of implementation https://gerrit.wikimedia.org/r/384047TASK

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 339575 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Add script to search entities from command line https://gerrit.wikimedia.org/r/339575TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 383464 had a related patch set uploaded (by Smalyshev; owner: Smalyshev): [operations/mediawiki-config@master] Add configuration for statement indexing for Wikidata https://gerrit.wikimedia.org/r/383464TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 383364 had a related patch set uploaded (by Thiemo Mättig (WMDE); owner: Thiemo Mättig (WMDE)): [mediawiki/extensions/WikibaseLexeme@master] Bind against FieldDefinitions interface instead of implementation https://gerrit.wikimedia.org/r/383364TASK

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 376645 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Enable indexing statements on items https://gerrit.wikimedia.org/r/376645TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 382725 had a related patch set uploaded (by Thiemo Mättig (WMDE); owner: Thiemo Mättig (WMDE)): [mediawiki/extensions/Wikibase@master] Optimize StatementsField for performance and readability https://gerrit.wikimedia.org/r/382725TASK

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 339575 had a related patch set uploaded (by Daniel Kinzler; owner: Smalyshev): [mediawiki/extensions/Wikibase@master] Add script to search entities from command line https://gerrit.wikimedia.org/r/339575TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. I've renamed it to statement_keywords. Hopefully it's better.TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: aude, gerritbot, debt, EBernhardson, dcausse, daniel,

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

dcausse added a comment. Right now the field name is statements. I'm not sure whether we should add wb there (everything in that index is "wb", since it's on wikidata). What do you mean by "typed" though? I mean a name that bears the data types it stores, for me "statements" seems too generic, if

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. Moving the filtering to the mapping (which I'll find more flexible in the future) will require some custom mapper/analyzer. Right. That's why I prefer to postpone it for now. It's not required for immediate use cases and we can always add it later. But for me the most

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

dcausse added a comment. I like the idea to bind the elastic property to the type of the statement. For now writing a mapping with default elastic tools allows to index nothing or everything, filtering must be done on the php side like you did in the current patch. Moving the filtering to the

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. In the patch, there was an option raised to index all statements of certain type, instead of just named properties. I am not sure yet whether it is a good idea or not, need some thought. Probably not in the initial iteration, but possibly later.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. I'm not sure we should really go as far as indexing all statements, now. Most of them would not be very useful for the search purposes for now, and already served by Query Service. Most useful ones would be those that are legitimately limit the searches for relevant

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

dcausse added a comment. maybe custom analysis components in the extra plugin would make this easier? Unless we have some objections to making wikibase dependent on the wmf elastic plugins?TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

EBernhardson added a comment. I suppose if we want to send all the properties to elasticsearch, but only have it index specific ones we can apply the keep words token filter to relationships.properties, i'm not seeing anything obvious for relationships itself. I thought pattern match might be able

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. @EBernhardson yes, this looks like what I've done in the patch, I just wondered if it's correct. Looks like it is then :)TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

EBernhardson added a comment. I think the analyzer was just pseudo code, to actually make it happen you need something like this: https://phabricator.wikimedia.org/P5975 That script outputs at the end { "relationships": [ "P1:Q1234", "P31:Q54321", "P31:Q7654" ],

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. @dcausse Could you explain a bit more how to set up the analyzer? I tried to figure how to do it but I'm not sure whether I did it right.TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

gerritbot added a comment. Change 376645 had a related patch set uploaded (by Smalyshev; owner: Smalyshev): [mediawiki/extensions/Wikibase@master] [WIP] Index statements on items https://gerrit.wikimedia.org/r/376645TASK DETAILhttps://phabricator.wikimedia.org/T175199EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

dcausse added a comment. deboosting can happen in the rescore stage, since we use a weighted sum we can either apply a negative penalty when relationship:P31:Q4167410 or a positive value when NOT relationship:P31:Q4167410. Will we add all properties or just a set of selected properties?

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. I wonder also, is it possible to do the (de)boosting on rescore stage? The reason is because we can select different rescore profiles from URL (which means different widgets can use different boosts) while getting stuff added to the search query itself is more

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

EBernhardson added a comment. In T175199#3585954, @Smalyshev wrote: i wonder if we could rather have some sort of relationship (name tbd) keyword field that encodes both parts That would depend on whether we could use such things for boosting/de-boosting. If yes, this certainly could be a way to

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

Smalyshev added a comment. i wonder if we could rather have some sort of relationship (name tbd) keyword field that encodes both parts That would depend on whether we could use such things for boosting/de-boosting. If yes, this certainly could be a way to go. That, however, makes it harder to do

[Wikidata-bugs] [Maniphest] [Commented On] T175199: Index certain statements for Wikidata items

EBernhardson added a comment. One worry i have is about over-creating fields. If we are talking about 5 relationships then maybe it's no big deal, but if we want to capture many different relationships, both in wikidata and eventually in structured data, i wonder if we could rather have some sort