[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-09 Thread dcausse 
dcausse closed this task as a duplicate of T145712: Use RDF statement counts 
from entity data, not page props ( wikibase:identifiers, wikibase:statements 
and wikibase:sitelinks ).

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Lucas_Werkmeister_WMDE, dcausse, Aklapper, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-02 Thread CBogen 
CBogen triaged this task as "High" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: CBogen
Cc: Lucas_Werkmeister_WMDE, dcausse, Aklapper, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-02 Thread DannyS712 
DannyS712 removed a parent task: T267016: Plan for november meeting with 
Göteborgs museer.

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: DannyS712
Cc: Lucas_Werkmeister_WMDE, dcausse, Aklapper, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-02 Thread Hermann 
Hermann added a parent task: T267016: Plan for november meeting with Göteborgs 
museer.

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Hermann
Cc: Lucas_Werkmeister_WMDE, dcausse, Aklapper, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-02 Thread dcausse 
dcausse added a comment.


  @Lucas_Werkmeister_WMDE indeed, thanks for the link I was not aware of this 
ticket! :)
  
  I think we agree that most of this data can be computed using the data 
available in the entity and not rely on page properties, the only one that 
remains difficult is the number of identifiers as it depends on the properties.

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Lucas_Werkmeister_WMDE, dcausse, Aklapper, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-02 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  @dcausse it sounds like we’re attacking the same problem from two different 
angles? See my recent comments in T145712 
 :)

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, dcausse, Aklapper, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-02 Thread dcausse 
dcausse added a parent task: T244590: [Epic] Rework the WDQS updater as an 
event driven application.

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-02 Thread dcausse 
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T267001: Compute page properties information at munge time

2020-11-02 Thread dcausse 
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
  As a maintainer of wdqs I want to workaround T266999 
 so that I can properly detect what 
changed on entity between two revisions instead of relying on the notion of 
//current// state.
  
  Some triples in the RDF output like page properties may not depend entirely 
on the data stored in the entity revision.
  This cause the issue that it is impossible to reconstruct the RDF output of 
an entity for a particular revision.
  To workaround these issue the munger could generate these values on the fly 
instead of relying on the ones stored in the page properties 
(`\Wikibase\Repo\Content\EntityContent::applyEntityPageProperties`):
  
  - Items:
- statements should be easy as it is the number of statement and can easily 
be counted reading the RDF output
- sitelinks is similar at it solely depends on the data of the entity itself
- identifiers: is more delicate as it depends on the type of the properties 
being used
  - Properties
- statements
  - Lexemes:
- statements
- senses: can be inferred from the entity content (not generated currently)
- forms: can be inferred from the entity content (not generated currently)
  
  Overall most of these values can be inferred directly from the entity content 
at munge time.
  
  Sole exception is the number of //identifiers// which requires the knowledge 
of which entity has a `wikibase:propertyType` equal to `wikibase:ExternalId` 
(https://w.wiki/jnk).
  Currently 5451 properties are identified as such and it should be possible to 
deploy small dataset within the deploy repo containing such information so that 
the munger can properly infer the number of identifiers at munge time.
  To make it stable this dataset will be //append only// and the date of 
generation will be matched against the modification data of the entity being 
munged.
  
  AC:
  
  - triples `wikibase:statements`, `wikibase:sitelinks` and 
`wikibase:identifiers` are ignored from the wikibase dumps and generated on the 
fly at munge time

TASK DETAIL
  https://phabricator.wikimedia.org/T267001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs