Hi all, I'd like to revive this discussion. This FLIP was originally proposed and went to a vote last year (it received two +1s) [1]. Since the original proposal and vote were about a year ago, I want to revisit the discussion before starting a fresh vote to ensure the design remains fully aligned with the current master branch.
At a high level, the proposal is unchanged from the previous round. The one substantive refinement since the earlier votes is the sampling approach detailed below, which directly addresses the performance overhead question Zakelly raised (the last open item on the thread). Overhead (Zakelly's question): Processing time is no longer measured on every invocation. It now uses the same counter-based sampling as Flink's state latency tracking (FLINK-21736). A new `table.exec.udf-metric.sample-interval` option (default 100) means only every Nth invocation is timed. The non-sampled fast path is a single integer increment, and `udfProcessingTime` is now a Histogram (p50/p75/p95/p99) backed by a bounded 128-entry circular buffer. Combined with the existing `table.exec.udf-metric-enabled` gate (off by default, meaning nothing is registered when disabled), this provides two robust layers of protection against performance overhead. - Proposal doc (updated): [2] - cwiki FLIP-485: [3] - Draft PR: [4] (implements the proposal; kept in draft until we converge) I welcome your renewed feedback and any questions. If there are no further concerns, I'll start a fresh [VOTE] thread on the updated proposal so the final votes reflect the current design. Thanks, Weiqing [1] Previous vote thread: https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3 [2] https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit [3] https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics [4] https://github.com/apache/flink/pull/28692 On Wed, Jun 3, 2026 at 4:59 PM Weiqing Yang <[email protected]> wrote: > Hi Zakelly, > > Kindly pinging here to see if you had any remaining concerns regarding the > FLIP. > > If there are no further questions or concerns from anyone, I plan to close > this discussion thread and proceed with the vote thread. > > Thanks, > Weiqing > > On Tue, Mar 3, 2026 at 2:58 PM Weiqing Yang <[email protected]> > wrote: > >> Hi Zakelly, >> >> >> Thanks for the feedback and sorry for the late response - I am now >> picking it back up. >> >> You raised a great point about the performance overhead, referencing >> FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>. I've >> updated the FLIP to adopt the same counter-based sampling approach used by >> Flink's state latency tracking (FLINK-21736 >> <https://issues.apache.org/jira/browse/FLINK-21736>). Specifically: >> >> 1. New config: table.exec.udf-metric.sample-interval (default: 100 [1]) >> - only every Nth invocation is measured >> 2. Fast path: Non-sampled invocations are a single integer increment - >> negligible overhead >> 3. Sampled path: System.nanoTime() around the UDF call, stored in a >> DescriptiveStatisticsHistogram with a bounded 128-entry circular buffer [2] >> 4. Metric type change: udfProcessingTime is now a Histogram (reports >> p50/p75/p95/p99/mean/min/max) instead of the original Gauge >> 5. Exception counting: Not sampled, since exceptions are rare events >> and counting each one has negligible cost >> >> Combined with the existing feature gate (table.exec.udf-metric-enabled >> defaulting to false), users have two layers of protection: the feature is >> off by default, and when enabled, sampling keeps overhead minimal. >> The updated FLIP is here: link >> <https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1> >> >> Would this address your concern? If so, it would be great to have your >> vote on the vote thread [3]. >> >> [1] 100: state.latency-track.sample-interval default value >> >> [2] 128: state.latency-track.history-size default value (line 55), which >> is the circular buffer size for the DescriptiveStatisticsHistogram >> [3] https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3 >> >> Thanks, >> Weiqing >> >> On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <[email protected]> >> wrote: >> >>> Hi Weiqing, >>> >>> Sorry for the late reply. And I have one question: >>> >>> I'm wondering whether the UDF processing time is measured for every >>> individual UDF invocation, with the average then reported, or if sampling >>> is used instead? I'm concerned about the potential overhead if we measure >>> every single invocation. We've encountered similar performance issues >>> when >>> implementing state latency tracking [1]. >>> >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-16444 >>> >>> Best, >>> Zakelly >>> >>> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <[email protected]> >>> wrote: >>> >>> > Cool - I’ll proceed to start the VOTE. >>> > Thanks! >>> > >>> > Weiqing >>> > >>> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <[email protected]> >>> wrote: >>> > >>> > > I don't have any more comments. >>> > > >>> > > Best, >>> > > Shengkai >>> > > >>> > > Weiqing Yang <[email protected]> 于2025年8月14日周四 14:47写道： >>> > > >>> > > > Thanks, Shengkai. I’ve updated the proposal doc with the >>> recommended >>> > > > configuration name. Please let me know if you have any additional >>> > > feedback. >>> > > > >>> > > > Best, >>> > > > Weiqing >>> > > > >>> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <[email protected]> >>> > wrote: >>> > > > >>> > > > > Sorry for the late response. I prefer to use >>> > > > > `table.exec.udf-metric-enabled` as the option name. >>> > > > > >>> > > > > Best, >>> > > > > Shengkai >>> > > > > >>> > > > > Weiqing Yang <[email protected]> 于2025年8月13日周三 23:54写道： >>> > > > > >>> > > > > > Hi Shengkai, Alan, Xuyang, and all, >>> > > > > > >>> > > > > > Since there have been no further objections, I’ll proceed to >>> start >>> > > the >>> > > > > VOTE >>> > > > > > on this proposal shortly. >>> > > > > > >>> > > > > > Thanks, >>> > > > > > Weiqing >>> > > > > > >>> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang < >>> > > > [email protected]> >>> > > > > > wrote: >>> > > > > > >>> > > > > > > Hi Shengkai, Alan and Xuyang, >>> > > > > > > >>> > > > > > > Just checking in - do you have any concerns or feedback? >>> > > > > > > >>> > > > > > > If there are no further objections from anyone, I’ll mark the >>> > FLIP >>> > > as >>> > > > > > > ready for voting. >>> > > > > > > >>> > > > > > > >>> > > > > > > Best, >>> > > > > > > Weiqing >>> > > > > > > >>> > > > > > > >>> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang < >>> > > > [email protected] >>> > > > > > >>> > > > > > > wrote: >>> > > > > > > >>> > > > > > >> Hi Xuyang, >>> > > > > > >> >>> > > > > > >> Thank you for reviewing the proposal! >>> > > > > > >> >>> > > > > > >> I’m planning to use: *udf.metrics.process-time* and >>> > > > > > >> *udf.metrics.exception-count*. These follow the naming >>> > convention >>> > > > used >>> > > > > > >> in Flink (e.g., RocksDB native metrics >>> > > > > > >> < >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics >>> > > > > > >). >>> > > > > > >> I’ve added these names to the proposal doc. >>> > > > > > >> >>> > > > > > >> Alternatively, I also considered: >>> > > *metrics.udf.process-time.enabled* >>> > > > > and >>> > > > > > >> *metrics.udf.exception-count.enabled. * >>> > > > > > >> >>> > > > > > >> Happy to hear any feedback on which style might be more >>> > > appropriate. >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> Best, >>> > > > > > >> Weiqing >>> > > > > > >> >>> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <[email protected]> >>> > > wrote: >>> > > > > > >> >>> > > > > > >>> Hi, Weiqing. >>> > > > > > >>> >>> > > > > > >>> Thanks for driving to improve this. I just have one >>> question. I >>> > > > > notice >>> > > > > > a >>> > > > > > >>> new configuration is introduced in this flip. I just wonder >>> > what >>> > > > the >>> > > > > > >>> configuration name is. Could you please include the full >>> name >>> > of >>> > > > this >>> > > > > > >>> configuration? (just similar to the other names in >>> > > MetricOptions?) >>> > > > > > >>> >>> > > > > > >>> >>> > > > > > >>> >>> > > > > > >>> >>> > > > > > >>> -- >>> > > > > > >>> >>> > > > > > >>> Best！ >>> > > > > > >>> Xuyang >>> > > > > > >>> >>> > > > > > >>> >>> > > > > > >>> >>> > > > > > >>> >>> > > > > > >>> >>> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" < >>> [email protected] >>> > > >>> > > > 写道： >>> > > > > > >>> >Hi Alan, >>> > > > > > >>> > >>> > > > > > >>> >Thanks for reviewing the proposal and for highlighting the >>> > > > > ASYNC_TABLE >>> > > > > > >>> work. >>> > > > > > >>> > >>> > > > > > >>> >Yes, I’ve updated the proposal to cover both ASYNC_SCALAR >>> and >>> > > > > > >>> ASYNC_TABLE. >>> > > > > > >>> >For async UDFs, the plan is to instrument both the >>> > invokeAsync() >>> > > > > call >>> > > > > > >>> and >>> > > > > > >>> >the async callback handler to measure the full end-to-end >>> > > latency >>> > > > > > until >>> > > > > > >>> the >>> > > > > > >>> >result or error is returned from the future. >>> > > > > > >>> > >>> > > > > > >>> >Let me know if you have any further questions or >>> suggestions. >>> > > > > > >>> > >>> > > > > > >>> >Best, >>> > > > > > >>> >Weiqing >>> > > > > > >>> > >>> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg >>> > > > > > >>> ><[email protected]> wrote: >>> > > > > > >>> > >>> > > > > > >>> >> Hi Weiqing, >>> > > > > > >>> >> >>> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the >>> codegen >>> > is >>> > > > > > >>> >> ExprCodeGenerator which should invoke >>> > > > BridgingSqlFunctionCallGen, >>> > > > > > >>> which >>> > > > > > >>> >> could be instrumented with metrics. This works well for >>> > > > > synchronous >>> > > > > > >>> calls, >>> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be merged >>> > > > ASYNC_TABLE >>> > > > > ( >>> > > > > > >>> >> https://github.com/apache/flink/pull/26567)? Timing >>> > metrics >>> > > > > would >>> > > > > > >>> only >>> > > > > > >>> >> account for what it takes to call invokeAsync, not for >>> the >>> > > > result >>> > > > > to >>> > > > > > >>> >> complete (with a result or error from the future >>> object). >>> > > > > > >>> >> >>> > > > > > >>> >> There are appropriate places which can handle the async >>> > > > callbacks, >>> > > > > > >>> but they >>> > > > > > >>> >> are in other locations. Will you be able to support >>> those >>> > as >>> > > > > well? >>> > > > > > >>> >> >>> > > > > > >>> >> Thanks, >>> > > > > > >>> >> Alan >>> > > > > > >>> >> >>> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang < >>> > > [email protected] >>> > > > > >>> > > > > > >>> wrote: >>> > > > > > >>> >> >>> > > > > > >>> >> > I just have some questions: >>> > > > > > >>> >> > >>> > > > > > >>> >> > 1. The current metrics hierarchy shows that the UDF >>> metric >>> > > > group >>> > > > > > >>> belongs >>> > > > > > >>> >> to >>> > > > > > >>> >> > the TaskMetricGroup. I think it would be better for >>> the >>> > UDF >>> > > > > metric >>> > > > > > >>> group >>> > > > > > >>> >> to >>> > > > > > >>> >> > belong to the OperatorMetricGroup instead, because a >>> UDF >>> > > might >>> > > > > be >>> > > > > > >>> used by >>> > > > > > >>> >> > multiple operators. >>> > > > > > >>> >> > 2. What are the naming conventions for UDF metrics? >>> Could >>> > > you >>> > > > > > >>> provide an >>> > > > > > >>> >> > example? Do the metric name contains the UDF name? >>> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric introduced? If >>> a >>> > UDF >>> > > > > throws >>> > > > > > >>> an >>> > > > > > >>> >> > exception, the job fails immediately. Why do we need >>> to >>> > > track >>> > > > > this >>> > > > > > >>> value? >>> > > > > > >>> >> > >>> > > > > > >>> >> > Best >>> > > > > > >>> >> > Shengkai >>> > > > > > >>> >> > >>> > > > > > >>> >> > >>> > > > > > >>> >> > Weiqing Yang <[email protected]> 于2025年7月9日周三 >>> > > 12:59写道： >>> > > > > > >>> >> > >>> > > > > > >>> >> > > Hi all, >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > I’d like to initiate a discussion about adding UDF >>> > > metrics. >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > *Motivation* >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > User-defined functions (UDFs) are essential for >>> custom >>> > > logic >>> > > > > in >>> > > > > > >>> Flink >>> > > > > > >>> >> > jobs >>> > > > > > >>> >> > > but often act as black boxes, making debugging and >>> > > > performance >>> > > > > > >>> tuning >>> > > > > > >>> >> > > difficult. When issues like high latency or frequent >>> > > > > exceptions >>> > > > > > >>> occur, >>> > > > > > >>> >> > it's >>> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs. >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > Flink currently lacks built-in metrics for key UDF >>> > aspects >>> > > > > such >>> > > > > > as >>> > > > > > >>> >> > > per-record processing time or exception count. This >>> > limits >>> > > > > > >>> >> observability >>> > > > > > >>> >> > > and complicates: >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > - Debugging production issues >>> > > > > > >>> >> > > - Performance tuning and resource allocation >>> > > > > > >>> >> > > - Supplying reliable signals to autoscaling >>> systems >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will >>> improve >>> > > > platform >>> > > > > > >>> >> > > observability and overall health. >>> > > > > > >>> >> > > Here’s the proposal document: Link >>> > > > > > >>> >> > > < >>> > > > > > >>> >> > > >>> > > > > > >>> >> > >>> > > > > > >>> >> >>> > > > > > >>> >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1 >>> > > > > > >>> >> > > > >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > Your feedback and ideas are welcome to refine this >>> > > feature. >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > >>> > > > > > >>> >> > > Thanks, >>> > > > > > >>> >> > > Weiqing >>> > > > > > >>> >> > > >>> > > > > > >>> >> > >>> > > > > > >>> >> >>> > > > > > >>> >>> > > > > > >> >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> >>
