More interested whether a StreamingFileSink without a drain negatively affects it's exactly-once semantics , given that I state on SP would have the offsets from kafka + the valid lengths of the part files at SP. To be honest not sure whether the flushed buffers on sink are included in the length, or this is not an issue with StreamingFileSink. If it is the former then I would assume we should be documented and then have to look why this hang happens.
On Mon, Mar 29, 2021 at 4:08 PM Vishal Santoshi <vishal.santo...@gmail.com> wrote: > Is this a known issue. We do a stop + savepoint with drain. I see no back > pressure on our operators. It essentially takes a SP and then the SInk ( > StreamingFileSink to S3 ) just stays in the RUNNING state. > > Without drain i stop + savepoint works fine. I would imagine drain is > important ( flush the buffers etc ) but why this hang ( I did it 3 times > and waited 15 minutes each time ). > > Regards. >
