zentol commented on pull request #18825: URL: https://github.com/apache/flink/pull/18825#issuecomment-1046722375
aha. The SinkWriterOperator passes data to the sinkWriter (which counts stuff as it writes them to Kafka), but then also sends a commitable to the co-located committer, which also of course counts as a record. So, question 1: This doesn't seem specific to Kafka but a fundamental flaw in how the SinkWriterOperator works. Why are you only addressing it for Kafka? Question 2: How did you guys think this should be exposed to the user? What you are running into is an unanswered question w.r.t. metrics, as you have 1 operator writing data to the outside while also writing data to the next operator. Now what is the numRecordsOut for that operator? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
