weiqingy opened a new pull request, #30044:
URL: https://github.com/apache/flink/pull/30044

   ## What is the purpose of the change
   
   This pull request refines where UDF metrics are registered when 
`table.exec.udf-metric-enabled` is true:
   
     - Only code that runs inside an operator registers metrics. Code generated 
for constant folding, partition pruning, watermark generators pushed into a 
source and `SpecializedFunction` expression evaluators is opened without a 
runtime context and skips registration, so these paths work with metrics 
enabled.
     - Only user-defined functions are metered, as documented. Built-in 
functions, including the internal functions used for `UNNEST` and `INTERSECT 
ALL`, do not get `udf.*` groups.
     - A function used in several parts of one operator (for example a 
correlate's arguments and condition, a lookup join's pre-filter and join 
condition, or `DEFINE` and `MEASURES` of `MATCH_RECOGNIZE`) registers once per 
operator, and all parts share one handle, so every call is reported.
   
   ## Brief change log
   
     - `CodeGeneratorContext#supportsUdfMetrics` is true by default and false 
for the contexts used for constant folding and partition pruning, for 
pushed-down watermark generators and for `SpecializedFunction` expression 
evaluators. Child contexts inherit it. 
`BridgingFunctionGenUtil#udfMetricsTermIfEnabled` generates no metrics where it 
is false.
     - `BridgingSqlFunctionCallGen` passes no metric name when the function 
definition is a `BuiltInFunctionDefinition`. This also covers the internal 
UNNEST and REPLICATE_ROWS functions.
     - `UdfMetrics.register` returns the handle already registered on the same 
`udf.<name>` group, so all generated classes of one operator record into one 
histogram and one counter. Handles are cached with weak keys compared by 
identity, so the group of a finished task is not kept reachable, and a 
restarted task registers on a new group and gets a new handle.
     - No docs change: the docs and the option description already say the 
metrics are registered for user-defined functions.
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - 
`ExpressionReductionRulesTest#testExpressionReductionWithUDFAndUdfMetricsEnabled`:
 a UDF call with constant arguments is folded with metrics on.
     - `PartitionableSourceTest#testWithUdfAndUdfMetricsEnabled`: a UDF in a 
partition filter plans and prunes with metrics on.
     - `UdfMetricsITCase#testFunctionCallingExpressionEvaluator`: a 
`SpecializedFunction` whose evaluator calls another UDF runs with metrics on.
     - `UdfMetricsITCase#testFunctionInPushedDownWatermark`: a user-defined 
function in a `WATERMARK` expression pushed into the source runs with metrics 
on.
     - `UdfMetricsITCase#testBuiltInFunctionsNotMetered`: `TYPEOF` and `UNNEST` 
register no group, while a user-defined function in the same query does.
     - `UdfMetricsITCase#testCorrelateInvocationAndConditionShareOneHandle` and 
`#testLookupJoinPreFilterAndConditionShareOneHandle`: every call is sampled and 
the exception is counted.
     - `UdfMetricsTest#restartedOperatorGetsNewHandle` and 
`#doesNotRetainOperatorMetricGroup`: the shared handle follows the operator 
group and does not keep it reachable.
     - Each new test was also run without its part of the change, to confirm it 
covers that part. The planner plan tests and `UdfMetricsITCase`, 
`AsyncCalcITCase`, `AsyncCorrelateITCase`, the lookup join, MATCH_RECOGNIZE and 
watermark push-down ITCases pass.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code 2.1.322 (Claude Opus 5.5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to