dcausse added a comment.
In T248449#6411584 <https://phabricator.wikimedia.org/T248449#6411584>, @Zbyszko wrote: > In T248449#6411542 <https://phabricator.wikimedia.org/T248449#6411542>, @dcausse wrote: > >> In T248449#6382230 <https://phabricator.wikimedia.org/T248449#6382230>, @Zbyszko wrote: >> >>> We need to decide our approach on possible data corruption issues as well. One that comes to mind is a rev create with revid higher than previouis page delete. >> >> All scenario could happen if events are received out of order and are not necessarily "data corruption", revcreate with revid > a previous page delete may happen if a page-undelete event for this revid is late. > > Right, I wasn't aware back when writing a comment how exactly undeletes are handled - probably in most case that will be out of order events. I'm wondering how should we treat those situation - if we won't receive some intermediate revision, we simply diff the newer one received and ignore the one out of order. Should we treat page undeletes events the same way? I don't know... I hope that for the most cases late events will properly be detected by the window function and thus logged accordingly in a side output. For other situations, serious bug in MW, changeprop failing to propagate a particular topic, machines with completely unsynced clocks or any other serious issue affecting the validity of the streams we should aim imho to detect these as much as we can and log to the spurious-events side output anything that we think is suspicious. I would not try (yet) to recover such inconsistencies but rather learn what are the most common problems and tune the stream accordingly. Baring any lost events I think we should be able to track all these problems, either because lateness or because inconsistent with the current state. TASK DETAIL https://phabricator.wikimedia.org/T248449 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: dcausse, Aklapper, Zbyszko, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
