dcausse added a comment.
It might be feasible but it's not clear yet how hard it would be. Listing possible solutions (from the least desirable to the ideal): - reloading the dump is obviously a way to mitigate the issue, we want to make reloads more frequent but given the time it takes to do them it's unlikely that we could do more frequent than a monthly rate. For the time being this the sole work-around we have. - trying to reconcile like the current updater: fetch the "current" version of an item and reconcile it with the graph, based on a list of item to update that we would have to be fetched from the links table/API (I'd avoid relying on query service itself). I'm not a big fan of this solution because it's really unclear what is the "current" version of an item. - Ideally we'd like to give an "identity" of the RDF projection of an item, the streaming updater is currently using the revision of the item but this is clearly not sufficient in this case since the same item revision may result in a different RDF output at different point in time. Sadly the only identity I can think of is quite huge since it would be the item revision + all its properties:revisions pairs. Storing such state and detecting the creation/update of a property with a P1921 <https://phabricator.wikimedia.org/P1921> might allow creating a sidestream that we could then use to generate diffs for all the entities affected by this change (the logic to do the formatting will have to be implemented in the new updater but hopefully it's not something that should be too hard). Then we'd have to prioritize the stream to make sure that entity updates take precedence over this kind of updates. We might need to add a new flavor to EntityData to extract all the property revisions (something between the default flavor and the dump one). Assuming we have enough space to store the additional states in the flink pipeline I believe this might be doable in a streaming fashion and still send minimal diffs to blazegraph. I probably missed a lot of details though :) TASK DETAIL https://phabricator.wikimedia.org/T258685 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: dcausse, Aklapper, Lucas_Werkmeister_WMDE, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
