Hi John and Wencong, Thanks for the reply!
It is nice that optional-2 can address the problem without affecting the existing source connectors as far as functionality is concerned. One potential concern with this approach is that it might increase the Flink runtime overhead by adding one more virtual functional call to the per-record runtime call stack. Since Java's default MaxInlineLevel is 12-18, I believe it is easy for an operator chain of 5+ operators to exceed this limit. In this case. And option-2 would incur one more virtual table lookup to produce each record. It is not clear how much this overhead would show up for jobs with a chain of lightweight operators. I am recently working on FLINK-30531 <https://issues.apache.org/jira/browse/FLINK-30531> to reduce runtime overhead which might be related to this discussion. In comparison to option-2, the option-3 provided in my earlier email would not add this extra overhead. I think it might be worthwhile to invest in the long-term performance (and simpler runtime infra) and pay for the short-term cost of deprecating this metric in SourceOperatorBase. What do you think? Regards, Dong On Thu, Jan 5, 2023 at 10:10 PM Wencong Liu <liuwencle...@163.com> wrote: > Hi, All > > > Thanks for the reply! > > > I think both John and Dong's opinions are reasonable. John's Suggestion 2 > is a good implementation. > It does not affect the existing source connectors, but also provides > support > for custom counter in the future implementation. > > > WDYT? > > > Best, > > Wencong Liu
