dcausse added a comment.
Thanks for the advise, most of the streaming process of this application is keyed indeed. I haven't thought about using the CEP api provided by flink it sounds very flexible. As for this particular task I'm hoping that recent flink 1.11 that unifies periodic and punctuated watermarks <https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_timestamps_watermarks.html#writing-watermarkgenerators> will help. I'm curious to see if checkpoint times are reasonable even when backfilling from these two streams (we just switched over from eqiad to codfw, so it's probably the right time to work on this issue to see how it behaves) TASK DETAIL https://phabricator.wikimedia.org/T258687 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, dcausse, Aklapper, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
