Weiqing Yang created FLINK-41585:
------------------------------------

             Summary: UDF metrics: planning fails for UDFs in partition 
filters, built-in functions are metered, and operators lose samples
                 Key: FLINK-41585
                 URL: https://issues.apache.org/jira/browse/FLINK-41585
             Project: Flink
          Issue Type: Bug
          Components: Table SQL / Planner, Table SQL / Runtime
    Affects Versions: 2.4.0
            Reporter: Weiqing Yang
            Assignee: Weiqing Yang
             Fix For: 2.4.0


Release testing of FLIP-485 (FLINK-41450) found three problems with 
{{table.exec.udf-metric-enabled}} set to true. They are all in UDF metrics 
registration, so they are fixed together in one pull request.

# Code evaluated outside a running operator calls {{getRuntimeContext()}}: 
planning fails when a UDF or a built-in such as {{STARTSWITH}} is in a 
partition filter, constant calls are no longer folded, and a 
{{SpecializedFunction}} evaluator fails the job.
# Built-in functions such as {{TYPEOF}} and {{ARRAY_CONTAINS}}, and the 
internal UNNEST function, get UDF metric groups.
# A UDF used in more than one part of one operator (correlate, lookup join, 
MATCH_RECOGNIZE) loses the samples of all but one part.

h3. 1. Code evaluated outside a running operator fails

*What happens*

With {{table.exec.udf-metric-enabled}} set to true, three kinds of evaluation 
outside a running operator break:

* *Partition pruning fails the query.* A filter on a partition column that 
calls a user-defined function, or a built-in such as {{STARTSWITH}}, fails at 
planning, in streaming and batch:
{noformat}
java.lang.IllegalStateException: The runtime context has not been initialized.
    at 
org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:51)
    at PartitionPruner$13.open(Unknown Source)
    at 
org.apache.flink.table.planner.plan.utils.PartitionPruner$.prunePartitions(PartitionPruner.scala:129)
    at 
org.apache.flink.table.planner.plan.rules.logical.PushPartitionIntoTableSourceScanRule.lambda$onMatch$3(PushPartitionIntoTableSourceScanRule.java:160)
{noformat}
With metrics disabled the same query plans and prunes to the matching partition.
* *Constant folding is skipped.* A call with constant arguments, such as 
{{myudf('abc')}}, is no longer folded. The planner logs a WARN with the same 
exception on every reduction attempt (6 to 13 times for one small query) and 
evaluates the call per row. The reducer gives up on the whole batch of 
constants, so other constants in the same projection, for example 
{{UPPER('xyz')}}, are not folded either.
* *A {{SpecializedFunction}} evaluator fails the job.* A user function whose 
{{specialize}} calls another user-defined function through 
{{SpecializedContext#createEvaluator}} fails at runtime with the same 
{{IllegalStateException}}, because the generated evaluator is opened without a 
runtime context ({{DefaultExpressionEvaluator.java:62}}). With metrics disabled 
the job runs.

*How to reproduce*

Reproduced on master in planner tests, using the {{values}} test connector with 
{{'partition-list' = 'p:A;p:B'}} for the partitioned table. {{myudf}} is a 
deterministic {{ScalarFunction}} that returns {{s.toUpperCase() + "!"}}.
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
-- part_src (a INT, p STRING) PARTITIONED BY (p)
EXPLAIN SELECT a, p FROM part_src WHERE myudf(p) = 'A!';        -- fails, see 
above
EXPLAIN SELECT a, p FROM part_src WHERE STARTSWITH(p, 'A');     -- fails the 
same way
EXPLAIN SELECT a, p FROM part_src WHERE UPPER(p) = 'A';         -- plans and 
prunes (control)
-- src (a INT, b STRING)
EXPLAIN SELECT myudf('abc') AS c, a FROM src;                   -- not folded, 
WARN logged
{code}
With metrics disabled, the first two plan with only partition {{p=A}} scanned, 
and {{myudf('abc')}} is folded to {{'ABC!'}}.

*Cause*

With metrics enabled, {{CodeGeneratorContext#addReusableUdfMetrics}} adds 
{{UdfMetrics.register(getRuntimeContext().getMetricGroup(), ...)}} to the 
generated {{open()}}. {{ExpressionReducer}} ({{ExpressionReducer.scala:111}}) 
and {{PartitionPruner}} ({{PartitionPruner.scala:129}}) call {{open()}} on 
generated functions while planning, and {{DefaultExpressionEvaluator}} opens 
its generated evaluator at runtime; none of them has a runtime context. The 
reducer catches the exception and leaves the expressions unreduced. The pruner 
does not catch it, so planning fails.

*Proposed fix*

Do not instrument calls generated for these paths. The planner paths use a 
{{ConstantCodeGeneratorContext}} ({{ExpressionReducer.scala:78}}, 
{{PartitionPruner.scala:96}}) and the evaluator uses an 
{{EvaluatorCodeGeneratorContext}} ({{BridgingFunctionGenUtil.scala:1002}}), so 
{{BridgingFunctionGenUtil#udfMetricsTermIfEnabled}} can skip the metrics for 
these two contexts. Plan-time evaluation is not part of the running job, so it 
should not report job metrics. FLINK-19112 drew the same line for function 
metric groups during optimization. Add planner tests for both paths with 
metrics enabled.

h3. 2. Built-in functions are metered

*What happens*

With {{table.exec.udf-metric-enabled}} set to true, built-in functions that 
have a runtime implementation class get a UDF metric group, as if they were 
user-defined functions. This includes the internal functions the planner adds 
for {{UNNEST}} and {{INTERSECT ALL}}, so a query that calls no function at all 
can get a group such as {{udf.$UNNEST_ROWS$1}}. The metrics docs and the option 
description say these metrics are registered for SQL/Table user-defined 
functions.

*How to reproduce*

Reproduced on master in a planner integration test with an in-memory metric 
reporter, streaming mode, sample interval 1:
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
-- src (id INT, arr ARRAY<INT>, s STRING)
INSERT INTO sink SELECT TYPEOF(id), ARRAY_CONTAINS(arr, id), UPPER(s) FROM src;
INSERT INTO sink2 SELECT id, x FROM src CROSS JOIN UNNEST(arr) AS T(x);
INSERT INTO sink3 SELECT id FROM src INTERSECT ALL SELECT id FROM src;
{code}
Registered groups:
{noformat}
Calc:      udf.TYPEOF, udf.ARRAY_CONTAINS   (no group for UPPER)
Correlate: udf.$UNNEST_ROWS$1
Correlate: udf.$REPLICATE_ROWS$1
{noformat}
With {{table.exec.udf-metric-enabled}} set to false, no group is registered. A 
user-defined function in the same query gets only its own group, as expected.

*Cause*

Built-in functions with a runtime class are generated through 
{{BridgingSqlFunctionCallGen}}, the same entry as user-defined functions, and 
that entry always passes a metric name 
({{BridgingSqlFunctionCallGen.scala:82}}). Built-ins that {{ExprCodeGenerator}} 
handles explicitly, such as {{UPPER}}, do not take this path. On release-2.4 
about 60 public built-ins are affected (scalar or table built-ins with a 
runtime class, for example the {{ARRAY_\*}} and {{MAP_\*}} functions, 
{{TYPEOF}} and {{STARTSWITH}}), plus the internal {{$UNNEST_ROWS$1}}, 
{{$UNNEST_ROWS_WITH_ORDINALITY$1}} and {{$REPLICATE_ROWS$1}}.

Besides the extra groups and the sampling cost, this makes problem 1 reachable 
with built-ins alone, for example {{STARTSWITH}} in a partition filter.

*Proposed fix*

Do not pass a metric name in {{BridgingSqlFunctionCallGen}} when the function 
definition is a {{BuiltInFunctionDefinition}}. That also covers the internal 
UNNEST and REPLICATE_ROWS functions. Add a test that a query using only 
built-ins registers no UDF metric group.

h3. 3. A UDF used in several parts of one operator loses samples

*What happens*

With {{table.exec.udf-metric-enabled}} set to true, a user-defined function 
that is called both in the arguments of a table function and in the condition 
of the same correlate reports only part of its calls. The second registration 
of {{udf.<name>}} on the operator's metric group is rejected, and every sample 
and exception recorded through it is lost:
{noformat}
Name collision: Group already contains a Metric with the name 
'udfProcessingTime'. Metric will not be reported.
Name collision: Group already contains a Metric with the name 
'udfExceptionCount'. Metric will not be reported.
{noformat}

*How to reproduce*

Reproduced on master in a planner integration test with an in-memory metric 
reporter, sample interval 1, 3 source rows. {{probetableudf}} emits each input 
twice.
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
INSERT INTO sink SELECT x FROM src, LATERAL TABLE(probetableudf(probeudf(id))) 
AS T(x) WHERE probeudf(x) > 0;
{code}
The plan has one {{Correlate}} with 
{{invocation=\[probetableudf(probeudf($cor0.id))\]}} and 
{{condition=\[>(probeudf($0), 0)\]}}.

|| Case || Expected {{udf.probeudf}} || Reported ||
| invocation and condition | 9 samples | 6 samples, plus the two WARNs above |
| invocation and condition, the function throws once on the invocation side | 6 
samples, 1 exception | 4 samples, 0 exceptions |
| invocation only | 3 samples | 3 samples |
| condition only | 6 samples | 6 samples |

Counters inside the functions confirm that all calls are made. Only the metrics 
are lost.

*Cause*

The correlate's condition is generated into a separate collector class with its 
own {{CodeGeneratorContext}} ({{CorrelateCodeGenerator.scala:196}}). 
{{CodeGeneratorContext#addReusableUdfMetrics}} registers each function name 
once per context, so the operator class and the collector class each call 
{{UdfMetrics.register}} for the same name on the same operator metric group. 
The collector opens first, and the operator's registration is the one rejected.

Other operators that generate several classes on one metric group lose samples 
the same way when one function appears in more than one of them. Reproduced on 
master with sample interval 1:

|| Operator || Query shape || Calls || Reported samples ||
| Lookup join | {{cudf}} in the pre-filter and in the remaining join condition 
| 5 | 2 |
| MATCH_RECOGNIZE | {{cudf}} in {{DEFINE}} and in {{MEASURES}} | 6 | 3 |

Multi-way joins and delta joins generate their conditions the same way (code 
reading, not run).

*Proposed fix*

Register each function name once per operator rather than once per generated 
class: {{UdfMetrics.register}} returns the existing handle when the same 
operator metric group already has one for that name, so every generated class 
of the operator records into one histogram and one counter. Add tests for the 
correlate and lookup join cases above.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to