Re: Questions in sink exactly once implementation

2018-10-13 Thread Hequn Cheng
Hi Henry, > 1. I have heard a idempotent way but I do not know how to implement it, would you please enlighten me about it by a example? It's a property of the result data. For example, you can overwrite old values with new ones using a primary key. > 2. If dirty data are *added* but not updated

Re: Questions in sink exactly once implementation

2018-10-13 Thread 徐涛
Hi Hequn, Thanks a lot for your response. I have a few questions about this topic. Would you please help me about it? 1. I have heard a idempotent way but I do not know how to implement it, would you please enlighten me about it by a example? 2. If dirty data are added but

Re: Questions in sink exactly once implementation

2018-10-12 Thread Hequn Cheng
Hi Henry, Yes, exactly once using atomic way is heavy for mysql. However, you don't have to buffer data if you choose option 2. You can simply overwrite old records with new ones if result data is idempotent and this way can also achieve exactly once. There is a document about End-to-End Exactly-O

Questions in sink exactly once implementation

2018-10-12 Thread 徐涛
Hi I am reading the book “Introduction to Apache Flink”, and in the book there mentions two ways to achieve sink exactly once: 1. The first way is to buffer all output at the sink and commit this atomically when the sink receives a checkpoint record. 2. The second way is