Weiqing Yang created FLINK-41585:
------------------------------------
Summary: UDF metrics: planning fails for UDFs in partition
filters, built-in functions are metered, and operators lose samples
Key: FLINK-41585
URL: https://issues.apache.org/jira/browse/FLINK-41585
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner, Table SQL / Runtime
Affects Versions: 2.4.0
Reporter: Weiqing Yang
Assignee: Weiqing Yang
Fix For: 2.4.0
Release testing of FLIP-485 (FLINK-41450) found three problems with
{{table.exec.udf-metric-enabled}} set to true. They are all in UDF metrics
registration, so they are fixed together in one pull request.
# Code evaluated outside a running operator calls {{getRuntimeContext()}}:
planning fails when a UDF or a built-in such as {{STARTSWITH}} is in a
partition filter, constant calls are no longer folded, and a
{{SpecializedFunction}} evaluator fails the job.
# Built-in functions such as {{TYPEOF}} and {{ARRAY_CONTAINS}}, and the
internal UNNEST function, get UDF metric groups.
# A UDF used in more than one part of one operator (correlate, lookup join,
MATCH_RECOGNIZE) loses the samples of all but one part.
h3. 1. Code evaluated outside a running operator fails
*What happens*
With {{table.exec.udf-metric-enabled}} set to true, three kinds of evaluation
outside a running operator break:
* *Partition pruning fails the query.* A filter on a partition column that
calls a user-defined function, or a built-in such as {{STARTSWITH}}, fails at
planning, in streaming and batch:
{noformat}
java.lang.IllegalStateException: The runtime context has not been initialized.
at
org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:51)
at PartitionPruner$13.open(Unknown Source)
at
org.apache.flink.table.planner.plan.utils.PartitionPruner$.prunePartitions(PartitionPruner.scala:129)
at
org.apache.flink.table.planner.plan.rules.logical.PushPartitionIntoTableSourceScanRule.lambda$onMatch$3(PushPartitionIntoTableSourceScanRule.java:160)
{noformat}
With metrics disabled the same query plans and prunes to the matching partition.
* *Constant folding is skipped.* A call with constant arguments, such as
{{myudf('abc')}}, is no longer folded. The planner logs a WARN with the same
exception on every reduction attempt (6 to 13 times for one small query) and
evaluates the call per row. The reducer gives up on the whole batch of
constants, so other constants in the same projection, for example
{{UPPER('xyz')}}, are not folded either.
* *A {{SpecializedFunction}} evaluator fails the job.* A user function whose
{{specialize}} calls another user-defined function through
{{SpecializedContext#createEvaluator}} fails at runtime with the same
{{IllegalStateException}}, because the generated evaluator is opened without a
runtime context ({{DefaultExpressionEvaluator.java:62}}). With metrics disabled
the job runs.
*How to reproduce*
Reproduced on master in planner tests, using the {{values}} test connector with
{{'partition-list' = 'p:A;p:B'}} for the partitioned table. {{myudf}} is a
deterministic {{ScalarFunction}} that returns {{s.toUpperCase() + "!"}}.
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
-- part_src (a INT, p STRING) PARTITIONED BY (p)
EXPLAIN SELECT a, p FROM part_src WHERE myudf(p) = 'A!'; -- fails, see
above
EXPLAIN SELECT a, p FROM part_src WHERE STARTSWITH(p, 'A'); -- fails the
same way
EXPLAIN SELECT a, p FROM part_src WHERE UPPER(p) = 'A'; -- plans and
prunes (control)
-- src (a INT, b STRING)
EXPLAIN SELECT myudf('abc') AS c, a FROM src; -- not folded,
WARN logged
{code}
With metrics disabled, the first two plan with only partition {{p=A}} scanned,
and {{myudf('abc')}} is folded to {{'ABC!'}}.
*Cause*
With metrics enabled, {{CodeGeneratorContext#addReusableUdfMetrics}} adds
{{UdfMetrics.register(getRuntimeContext().getMetricGroup(), ...)}} to the
generated {{open()}}. {{ExpressionReducer}} ({{ExpressionReducer.scala:111}})
and {{PartitionPruner}} ({{PartitionPruner.scala:129}}) call {{open()}} on
generated functions while planning, and {{DefaultExpressionEvaluator}} opens
its generated evaluator at runtime; none of them has a runtime context. The
reducer catches the exception and leaves the expressions unreduced. The pruner
does not catch it, so planning fails.
*Proposed fix*
Do not instrument calls generated for these paths. The planner paths use a
{{ConstantCodeGeneratorContext}} ({{ExpressionReducer.scala:78}},
{{PartitionPruner.scala:96}}) and the evaluator uses an
{{EvaluatorCodeGeneratorContext}} ({{BridgingFunctionGenUtil.scala:1002}}), so
{{BridgingFunctionGenUtil#udfMetricsTermIfEnabled}} can skip the metrics for
these two contexts. Plan-time evaluation is not part of the running job, so it
should not report job metrics. FLINK-19112 drew the same line for function
metric groups during optimization. Add planner tests for both paths with
metrics enabled.
h3. 2. Built-in functions are metered
*What happens*
With {{table.exec.udf-metric-enabled}} set to true, built-in functions that
have a runtime implementation class get a UDF metric group, as if they were
user-defined functions. This includes the internal functions the planner adds
for {{UNNEST}} and {{INTERSECT ALL}}, so a query that calls no function at all
can get a group such as {{udf.$UNNEST_ROWS$1}}. The metrics docs and the option
description say these metrics are registered for SQL/Table user-defined
functions.
*How to reproduce*
Reproduced on master in a planner integration test with an in-memory metric
reporter, streaming mode, sample interval 1:
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
-- src (id INT, arr ARRAY<INT>, s STRING)
INSERT INTO sink SELECT TYPEOF(id), ARRAY_CONTAINS(arr, id), UPPER(s) FROM src;
INSERT INTO sink2 SELECT id, x FROM src CROSS JOIN UNNEST(arr) AS T(x);
INSERT INTO sink3 SELECT id FROM src INTERSECT ALL SELECT id FROM src;
{code}
Registered groups:
{noformat}
Calc: udf.TYPEOF, udf.ARRAY_CONTAINS (no group for UPPER)
Correlate: udf.$UNNEST_ROWS$1
Correlate: udf.$REPLICATE_ROWS$1
{noformat}
With {{table.exec.udf-metric-enabled}} set to false, no group is registered. A
user-defined function in the same query gets only its own group, as expected.
*Cause*
Built-in functions with a runtime class are generated through
{{BridgingSqlFunctionCallGen}}, the same entry as user-defined functions, and
that entry always passes a metric name
({{BridgingSqlFunctionCallGen.scala:82}}). Built-ins that {{ExprCodeGenerator}}
handles explicitly, such as {{UPPER}}, do not take this path. On release-2.4
about 60 public built-ins are affected (scalar or table built-ins with a
runtime class, for example the {{ARRAY_\*}} and {{MAP_\*}} functions,
{{TYPEOF}} and {{STARTSWITH}}), plus the internal {{$UNNEST_ROWS$1}},
{{$UNNEST_ROWS_WITH_ORDINALITY$1}} and {{$REPLICATE_ROWS$1}}.
Besides the extra groups and the sampling cost, this makes problem 1 reachable
with built-ins alone, for example {{STARTSWITH}} in a partition filter.
*Proposed fix*
Do not pass a metric name in {{BridgingSqlFunctionCallGen}} when the function
definition is a {{BuiltInFunctionDefinition}}. That also covers the internal
UNNEST and REPLICATE_ROWS functions. Add a test that a query using only
built-ins registers no UDF metric group.
h3. 3. A UDF used in several parts of one operator loses samples
*What happens*
With {{table.exec.udf-metric-enabled}} set to true, a user-defined function
that is called both in the arguments of a table function and in the condition
of the same correlate reports only part of its calls. The second registration
of {{udf.<name>}} on the operator's metric group is rejected, and every sample
and exception recorded through it is lost:
{noformat}
Name collision: Group already contains a Metric with the name
'udfProcessingTime'. Metric will not be reported.
Name collision: Group already contains a Metric with the name
'udfExceptionCount'. Metric will not be reported.
{noformat}
*How to reproduce*
Reproduced on master in a planner integration test with an in-memory metric
reporter, sample interval 1, 3 source rows. {{probetableudf}} emits each input
twice.
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
INSERT INTO sink SELECT x FROM src, LATERAL TABLE(probetableudf(probeudf(id)))
AS T(x) WHERE probeudf(x) > 0;
{code}
The plan has one {{Correlate}} with
{{invocation=\[probetableudf(probeudf($cor0.id))\]}} and
{{condition=\[>(probeudf($0), 0)\]}}.
|| Case || Expected {{udf.probeudf}} || Reported ||
| invocation and condition | 9 samples | 6 samples, plus the two WARNs above |
| invocation and condition, the function throws once on the invocation side | 6
samples, 1 exception | 4 samples, 0 exceptions |
| invocation only | 3 samples | 3 samples |
| condition only | 6 samples | 6 samples |
Counters inside the functions confirm that all calls are made. Only the metrics
are lost.
*Cause*
The correlate's condition is generated into a separate collector class with its
own {{CodeGeneratorContext}} ({{CorrelateCodeGenerator.scala:196}}).
{{CodeGeneratorContext#addReusableUdfMetrics}} registers each function name
once per context, so the operator class and the collector class each call
{{UdfMetrics.register}} for the same name on the same operator metric group.
The collector opens first, and the operator's registration is the one rejected.
Other operators that generate several classes on one metric group lose samples
the same way when one function appears in more than one of them. Reproduced on
master with sample interval 1:
|| Operator || Query shape || Calls || Reported samples ||
| Lookup join | {{cudf}} in the pre-filter and in the remaining join condition
| 5 | 2 |
| MATCH_RECOGNIZE | {{cudf}} in {{DEFINE}} and in {{MEASURES}} | 6 | 3 |
Multi-way joins and delta joins generate their conditions the same way (code
reading, not run).
*Proposed fix*
Register each function name once per operator rather than once per generated
class: {{UdfMetrics.register}} returns the existing handle when the same
operator metric group already has one for that name, so every generated class
of the operator records into one histogram and one counter. Add tests for the
correlate and lookup join cases above.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)