| Addshore added a comment. |
In T214362#4954428, @Joe wrote:In order to better understand your needs, let me ask you a few questions:
- Do we need/want just the constraint check for the latest version of the item, or one for each revision?
Currently there is only the need to store the latest constraint check data for an item.
- How will we access such constraints? Always by key and/or full dump, or other access patterns can become useful/interesting in the future?
There are currently no other access patterns on the horizon.
(storing the data like this will allow us to load it into the WDQS and query it from there)
- Given those values will only be updated via the jobqueue, we don't need active-active write capabilities in the storage, or you still want to be able to compute the check on-demand and thus a/a storage is recommendable?
We currently still want to be able to compute the check on demand, either because the user wants to purge the current constraint check data, or if the check data does not already exist / is outdated.
It could be possible that later down the line we put the purging of this data into the job queue too, and once we have data for all items persistently stored in theory the user would never ask for an items constraint check and it not be there (thus no writing to the storage on request)
But that is not the immediate plan.
I'm not sure how the "full dump" would need to work, but it would seem natural that such data would be fed to wdqs with the same mechanism that updates the items.
The main regular updates for the WDQS will come via events in kafka, and then the WDQS retrieving the constraint check data from an MW API in the same way that it retrieves changes to entities.
The "full dump" is needed when starting a WDQS server off from a fresh start.
The full dump could just be a PHP script that iterates through the storage for all entities and slowly dumps the data to disk (similar to our dumpJson or dumpRdf scripts, and similar to a regular MW dump)
Cc: Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, Nandana, kostajh, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, merbst, LawExplorer, _jensen, D3r1ck01, SBisson, Eevans, mobrovac, Hardikj, Wikidata-bugs, aude, GWicke, jayvdb, fbstj, santhosh, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Ltrlg, bd808, Legoktm
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
