Hi all, Gentle bump on this.
The proposal is unchanged at a high level. It's been re-aligned with the current master, and the earlier overhead concern is addressed (details in my previous message). Shengkai, Alan, Xuyang, Zakelly — thank you for the thorough review last time. Yuepeng and Shengkai, thank you for your votes on the earlier round. Since you already know this proposal well, I'd really appreciate a quick check that the points you raised still sit right in the current version. If there are no other concerns over the next few days, I'll start a fresh [VOTE] thread so the votes reflect the current design. FLIP doc: https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit Best, Weiqing On Thu, Jul 9, 2026 at 7:20 PM Weiqing Yang <[email protected]> wrote: > Hi all, > > I'd like to revive this discussion. This FLIP was originally proposed and > went to a vote last year (it received two +1s) [1]. Since the original > proposal and vote were about a year ago, I want to revisit the discussion > before starting a fresh vote to ensure the design remains fully aligned > with the current master branch. > > At a high level, the proposal is unchanged from the previous round. The > one substantive refinement since the earlier votes is the sampling approach > detailed below, which directly addresses the performance overhead question > Zakelly raised (the last open item on the thread). > > Overhead (Zakelly's question): Processing time is no longer measured on > every invocation. It now uses the same counter-based sampling as Flink's > state latency tracking (FLINK-21736). A new > `table.exec.udf-metric.sample-interval` option (default 100) means only > every Nth invocation is timed. The non-sampled fast path is a single > integer increment, and `udfProcessingTime` is now a Histogram > (p50/p75/p95/p99) backed by a bounded 128-entry circular buffer. Combined > with the existing `table.exec.udf-metric-enabled` gate (off by default, > meaning nothing is registered when disabled), this provides two robust > layers of protection against performance overhead. > > - Proposal doc (updated): [2] > > - cwiki FLIP-485: [3] > > - Draft PR: [4] (implements the proposal; kept in draft until we converge) > > I welcome your renewed feedback and any questions. If there are no further > concerns, I'll start a fresh [VOTE] thread on the updated proposal so the > final votes reflect the current design. > > Thanks, > > Weiqing > > [1] Previous vote thread: > https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3 > > [2] > https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit > > [3] > https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics > > [4] https://github.com/apache/flink/pull/28692 > > > On Wed, Jun 3, 2026 at 4:59 PM Weiqing Yang <[email protected]> > wrote: > >> Hi Zakelly, >> >> Kindly pinging here to see if you had any remaining concerns regarding >> the FLIP. >> >> If there are no further questions or concerns from anyone, I plan to >> close this discussion thread and proceed with the vote thread. >> >> Thanks, >> Weiqing >> >> On Tue, Mar 3, 2026 at 2:58 PM Weiqing Yang <[email protected]> >> wrote: >> >>> Hi Zakelly, >>> >>> >>> Thanks for the feedback and sorry for the late response - I am now >>> picking it back up. >>> >>> You raised a great point about the performance overhead, referencing >>> FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>. I've >>> updated the FLIP to adopt the same counter-based sampling approach used by >>> Flink's state latency tracking (FLINK-21736 >>> <https://issues.apache.org/jira/browse/FLINK-21736>). Specifically: >>> >>> 1. New config: table.exec.udf-metric.sample-interval (default: 100 >>> [1]) - only every Nth invocation is measured >>> 2. Fast path: Non-sampled invocations are a single integer increment - >>> negligible overhead >>> 3. Sampled path: System.nanoTime() around the UDF call, stored in a >>> DescriptiveStatisticsHistogram with a bounded 128-entry circular buffer [2] >>> 4. Metric type change: udfProcessingTime is now a Histogram (reports >>> p50/p75/p95/p99/mean/min/max) instead of the original Gauge >>> 5. Exception counting: Not sampled, since exceptions are rare events >>> and counting each one has negligible cost >>> >>> Combined with the existing feature gate (table.exec.udf-metric-enabled >>> defaulting to false), users have two layers of protection: the feature is >>> off by default, and when enabled, sampling keeps overhead minimal. >>> The updated FLIP is here: link >>> <https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1> >>> >>> Would this address your concern? If so, it would be great to have your >>> vote on the vote thread [3]. >>> >>> [1] 100: state.latency-track.sample-interval default value >>> >>> [2] 128: state.latency-track.history-size default value (line 55), which >>> is the circular buffer size for the DescriptiveStatisticsHistogram >>> [3] https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3 >>> >>> Thanks, >>> Weiqing >>> >>> On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <[email protected]> >>> wrote: >>> >>>> Hi Weiqing, >>>> >>>> Sorry for the late reply. And I have one question: >>>> >>>> I'm wondering whether the UDF processing time is measured for every >>>> individual UDF invocation, with the average then reported, or if >>>> sampling >>>> is used instead? I'm concerned about the potential overhead if we >>>> measure >>>> every single invocation. We've encountered similar performance issues >>>> when >>>> implementing state latency tracking [1]. >>>> >>>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-16444 >>>> >>>> Best, >>>> Zakelly >>>> >>>> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <[email protected]> >>>> wrote: >>>> >>>> > Cool - I’ll proceed to start the VOTE. >>>> > Thanks! >>>> > >>>> > Weiqing >>>> > >>>> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <[email protected]> >>>> wrote: >>>> > >>>> > > I don't have any more comments. >>>> > > >>>> > > Best, >>>> > > Shengkai >>>> > > >>>> > > Weiqing Yang <[email protected]> 于2025年8月14日周四 14:47写道： >>>> > > >>>> > > > Thanks, Shengkai. I’ve updated the proposal doc with the >>>> recommended >>>> > > > configuration name. Please let me know if you have any additional >>>> > > feedback. >>>> > > > >>>> > > > Best, >>>> > > > Weiqing >>>> > > > >>>> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <[email protected]> >>>> > wrote: >>>> > > > >>>> > > > > Sorry for the late response. I prefer to use >>>> > > > > `table.exec.udf-metric-enabled` as the option name. >>>> > > > > >>>> > > > > Best, >>>> > > > > Shengkai >>>> > > > > >>>> > > > > Weiqing Yang <[email protected]> 于2025年8月13日周三 23:54写道： >>>> > > > > >>>> > > > > > Hi Shengkai, Alan, Xuyang, and all, >>>> > > > > > >>>> > > > > > Since there have been no further objections, I’ll proceed to >>>> start >>>> > > the >>>> > > > > VOTE >>>> > > > > > on this proposal shortly. >>>> > > > > > >>>> > > > > > Thanks, >>>> > > > > > Weiqing >>>> > > > > > >>>> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang < >>>> > > > [email protected]> >>>> > > > > > wrote: >>>> > > > > > >>>> > > > > > > Hi Shengkai, Alan and Xuyang, >>>> > > > > > > >>>> > > > > > > Just checking in - do you have any concerns or feedback? >>>> > > > > > > >>>> > > > > > > If there are no further objections from anyone, I’ll mark >>>> the >>>> > FLIP >>>> > > as >>>> > > > > > > ready for voting. >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > Best, >>>> > > > > > > Weiqing >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang < >>>> > > > [email protected] >>>> > > > > > >>>> > > > > > > wrote: >>>> > > > > > > >>>> > > > > > >> Hi Xuyang, >>>> > > > > > >> >>>> > > > > > >> Thank you for reviewing the proposal! >>>> > > > > > >> >>>> > > > > > >> I’m planning to use: *udf.metrics.process-time* and >>>> > > > > > >> *udf.metrics.exception-count*. These follow the naming >>>> > convention >>>> > > > used >>>> > > > > > >> in Flink (e.g., RocksDB native metrics >>>> > > > > > >> < >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics >>>> > > > > > >). >>>> > > > > > >> I’ve added these names to the proposal doc. >>>> > > > > > >> >>>> > > > > > >> Alternatively, I also considered: >>>> > > *metrics.udf.process-time.enabled* >>>> > > > > and >>>> > > > > > >> *metrics.udf.exception-count.enabled. * >>>> > > > > > >> >>>> > > > > > >> Happy to hear any feedback on which style might be more >>>> > > appropriate. >>>> > > > > > >> >>>> > > > > > >> >>>> > > > > > >> Best, >>>> > > > > > >> Weiqing >>>> > > > > > >> >>>> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <[email protected] >>>> > >>>> > > wrote: >>>> > > > > > >> >>>> > > > > > >>> Hi, Weiqing. >>>> > > > > > >>> >>>> > > > > > >>> Thanks for driving to improve this. I just have one >>>> question. I >>>> > > > > notice >>>> > > > > > a >>>> > > > > > >>> new configuration is introduced in this flip. I just >>>> wonder >>>> > what >>>> > > > the >>>> > > > > > >>> configuration name is. Could you please include the full >>>> name >>>> > of >>>> > > > this >>>> > > > > > >>> configuration? (just similar to the other names in >>>> > > MetricOptions?) >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> -- >>>> > > > > > >>> >>>> > > > > > >>> Best！ >>>> > > > > > >>> Xuyang >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> >>>> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" < >>>> [email protected] >>>> > > >>>> > > > 写道： >>>> > > > > > >>> >Hi Alan, >>>> > > > > > >>> > >>>> > > > > > >>> >Thanks for reviewing the proposal and for highlighting >>>> the >>>> > > > > ASYNC_TABLE >>>> > > > > > >>> work. >>>> > > > > > >>> > >>>> > > > > > >>> >Yes, I’ve updated the proposal to cover both >>>> ASYNC_SCALAR and >>>> > > > > > >>> ASYNC_TABLE. >>>> > > > > > >>> >For async UDFs, the plan is to instrument both the >>>> > invokeAsync() >>>> > > > > call >>>> > > > > > >>> and >>>> > > > > > >>> >the async callback handler to measure the full end-to-end >>>> > > latency >>>> > > > > > until >>>> > > > > > >>> the >>>> > > > > > >>> >result or error is returned from the future. >>>> > > > > > >>> > >>>> > > > > > >>> >Let me know if you have any further questions or >>>> suggestions. >>>> > > > > > >>> > >>>> > > > > > >>> >Best, >>>> > > > > > >>> >Weiqing >>>> > > > > > >>> > >>>> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg >>>> > > > > > >>> ><[email protected]> wrote: >>>> > > > > > >>> > >>>> > > > > > >>> >> Hi Weiqing, >>>> > > > > > >>> >> >>>> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the >>>> codegen >>>> > is >>>> > > > > > >>> >> ExprCodeGenerator which should invoke >>>> > > > BridgingSqlFunctionCallGen, >>>> > > > > > >>> which >>>> > > > > > >>> >> could be instrumented with metrics. This works well >>>> for >>>> > > > > synchronous >>>> > > > > > >>> calls, >>>> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be merged >>>> > > > ASYNC_TABLE >>>> > > > > ( >>>> > > > > > >>> >> https://github.com/apache/flink/pull/26567)? Timing >>>> > metrics >>>> > > > > would >>>> > > > > > >>> only >>>> > > > > > >>> >> account for what it takes to call invokeAsync, not for >>>> the >>>> > > > result >>>> > > > > to >>>> > > > > > >>> >> complete (with a result or error from the future >>>> object). >>>> > > > > > >>> >> >>>> > > > > > >>> >> There are appropriate places which can handle the async >>>> > > > callbacks, >>>> > > > > > >>> but they >>>> > > > > > >>> >> are in other locations. Will you be able to support >>>> those >>>> > as >>>> > > > > well? >>>> > > > > > >>> >> >>>> > > > > > >>> >> Thanks, >>>> > > > > > >>> >> Alan >>>> > > > > > >>> >> >>>> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang < >>>> > > [email protected] >>>> > > > > >>>> > > > > > >>> wrote: >>>> > > > > > >>> >> >>>> > > > > > >>> >> > I just have some questions: >>>> > > > > > >>> >> > >>>> > > > > > >>> >> > 1. The current metrics hierarchy shows that the UDF >>>> metric >>>> > > > group >>>> > > > > > >>> belongs >>>> > > > > > >>> >> to >>>> > > > > > >>> >> > the TaskMetricGroup. I think it would be better for >>>> the >>>> > UDF >>>> > > > > metric >>>> > > > > > >>> group >>>> > > > > > >>> >> to >>>> > > > > > >>> >> > belong to the OperatorMetricGroup instead, because a >>>> UDF >>>> > > might >>>> > > > > be >>>> > > > > > >>> used by >>>> > > > > > >>> >> > multiple operators. >>>> > > > > > >>> >> > 2. What are the naming conventions for UDF metrics? >>>> Could >>>> > > you >>>> > > > > > >>> provide an >>>> > > > > > >>> >> > example? Do the metric name contains the UDF name? >>>> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric introduced? >>>> If a >>>> > UDF >>>> > > > > throws >>>> > > > > > >>> an >>>> > > > > > >>> >> > exception, the job fails immediately. Why do we need >>>> to >>>> > > track >>>> > > > > this >>>> > > > > > >>> value? >>>> > > > > > >>> >> > >>>> > > > > > >>> >> > Best >>>> > > > > > >>> >> > Shengkai >>>> > > > > > >>> >> > >>>> > > > > > >>> >> > >>>> > > > > > >>> >> > Weiqing Yang <[email protected]> 于2025年7月9日周三 >>>> > > 12:59写道： >>>> > > > > > >>> >> > >>>> > > > > > >>> >> > > Hi all, >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > I’d like to initiate a discussion about adding UDF >>>> > > metrics. >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > *Motivation* >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > User-defined functions (UDFs) are essential for >>>> custom >>>> > > logic >>>> > > > > in >>>> > > > > > >>> Flink >>>> > > > > > >>> >> > jobs >>>> > > > > > >>> >> > > but often act as black boxes, making debugging and >>>> > > > performance >>>> > > > > > >>> tuning >>>> > > > > > >>> >> > > difficult. When issues like high latency or >>>> frequent >>>> > > > > exceptions >>>> > > > > > >>> occur, >>>> > > > > > >>> >> > it's >>>> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs. >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > Flink currently lacks built-in metrics for key UDF >>>> > aspects >>>> > > > > such >>>> > > > > > as >>>> > > > > > >>> >> > > per-record processing time or exception count. This >>>> > limits >>>> > > > > > >>> >> observability >>>> > > > > > >>> >> > > and complicates: >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > - Debugging production issues >>>> > > > > > >>> >> > > - Performance tuning and resource allocation >>>> > > > > > >>> >> > > - Supplying reliable signals to autoscaling >>>> systems >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will >>>> improve >>>> > > > platform >>>> > > > > > >>> >> > > observability and overall health. >>>> > > > > > >>> >> > > Here’s the proposal document: Link >>>> > > > > > >>> >> > > < >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > >>>> > > > > > >>> >> >>>> > > > > > >>> >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1 >>>> > > > > > >>> >> > > > >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > Your feedback and ideas are welcome to refine this >>>> > > feature. >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > > Thanks, >>>> > > > > > >>> >> > > Weiqing >>>> > > > > > >>> >> > > >>>> > > > > > >>> >> > >>>> > > > > > >>> >> >>>> > > > > > >>> >>>> > > > > > >> >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> >>>
