Hi devs: I'd like to start a discussion about FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store[1]. In the whole data stream processing, there are consistency problems such as how to manage the dependencies of multiple jobs and tables, how to define and handle E2E delays, and how to ensure the data consistency of queries on flowing data? This FLIP aims to support data consistency and answer these questions.
I'v discussed the details of this FLIP with @Jingsong Lee and @libenchao offline several times. We hope to support data consistency of queries on tables, managing relationships between Flink jobs and tables and revising tables on streaming in Flink and Table Store to improve the whole data stream processing. Looking forward to your feedback. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-276%3A+Data+Consistency+of+Streaming+and+Batch+ETL+in+Flink+and+Table+Store Best, Shammon
