dcausse created this task. dcausse added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION FlinkKafkaConsumer <https://nightlies.apache.org/flink/flink-docs-release-1.14/release-notes/flink-1.14/#deprecate-flinkkafkaconsumer> and FlinkKafkaProducer <https://nightlies.apache.org/flink/flink-docs-release-1.14/release-notes/flink-1.14/#port-kafkasink-to-new-unified-sink-api-flip-143> are now deprecated, KafkaSource and KafkaSink should be used instead. Unfortunately as these operators do store a state some migration steps have to be respected: - the job must be stopped with a savepoint and offsets committed to kafka - the job must be started from this savepoint with most probably `--allowNonRestoredState` set to true Given the risk of failures it might be wise to do this in two steps: - add support for KafkaSink/KafkaSource in the codebase with an option to flip between the two implemations - drop usage of FlinkKafkaConsumer and FlinkKafkaProducer once the job is proven to run properly with KafkaSink/KafkaSource AC: - FlinkKafkaConsumer and FlinkKafkaProducer usage are dropped from the codebase TASK DETAIL https://phabricator.wikimedia.org/T326914 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: dcausse, Aklapper, AWesterinen, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles
_______________________________________________ Wikidata-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected]
