[
https://issues.apache.org/jira/browse/FLINK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiqing Yang updated FLINK-41585:
---------------------------------
Description:
With {{table.exec.udf-metric-enabled}} set to true:
# *Evaluation outside a running operator fails.* Code generated for constant
folding, partition pruning, watermark generators pushed into a source and
{{SpecializedFunction}} expression evaluators registers metrics, but is opened
without a runtime context. A user-defined function or {{STARTSWITH}} in a
partition filter fails planning with {{IllegalStateException: The runtime
context has not been initialized}}. Calls with constant arguments are not
folded. A user-defined function in a pushed-down {{WATERMARK}} expression, or
called by a {{SpecializedFunction}} evaluator, fails the job at runtime.
# *Built-in functions get UDF metric groups.* Built-ins with a runtime class,
such as {{TYPEOF}} and {{ARRAY_CONTAINS}}, and the internal functions used for
{{UNNEST}} and {{INTERSECT ALL}}, register {{udf.*}} groups.
# *Calls are not reported when a function is used in several parts of one
operator.* For example in a correlate's table function arguments and its
condition, in a lookup join's pre-filter and join condition, or in {{DEFINE}}
and {{MEASURES}} of {{MATCH_RECOGNIZE}}. Each part registers {{udf.<name>}} on
the same operator metric group, only the first registration is kept, and a
{{Name collision}} WARN is logged.
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
-- 1: fails planning (part_src is partitioned by p)
SELECT a FROM part_src WHERE myudf(p) = 'A';
-- 2: registers udf.TYPEOF
SELECT TYPEOF(id) FROM src;
-- 3: some calls of probeudf are not reported
SELECT x FROM src, LATERAL TABLE(tf(probeudf(id))) AS T(x) WHERE probeudf(x) >
0;
{code}
*Proposed fix*
# Do not register metrics in code generated for constant folding, partition
pruning, pushed-down watermark generators and expression evaluators.
# Do not register metrics for built-in functions.
# Register each function once per operator metric group, and share it across
the operator's generated classes.
was:
Release testing of FLIP-485 (FLINK-41450) found three problems with
{{table.exec.udf-metric-enabled}} set to true. They are all in UDF metrics
registration, so they are fixed together in one pull request.
# Code evaluated outside a running operator calls {{getRuntimeContext()}}:
planning fails when a UDF or a built-in such as {{STARTSWITH}} is in a
partition filter, constant calls are no longer folded, and a
{{SpecializedFunction}} evaluator fails the job.
# Built-in functions such as {{TYPEOF}} and {{ARRAY_CONTAINS}}, and the
internal UNNEST function, get UDF metric groups.
# A UDF used in more than one part of one operator (correlate, lookup join,
MATCH_RECOGNIZE) loses the samples of all but one part.
h3. 1. Code evaluated outside a running operator fails
*What happens*
With {{table.exec.udf-metric-enabled}} set to true, three kinds of evaluation
outside a running operator break:
* *Partition pruning fails the query.* A filter on a partition column that
calls a user-defined function, or a built-in such as {{STARTSWITH}}, fails at
planning, in streaming and batch:
{noformat}
java.lang.IllegalStateException: The runtime context has not been initialized.
at
org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:51)
at PartitionPruner$13.open(Unknown Source)
at
org.apache.flink.table.planner.plan.utils.PartitionPruner$.prunePartitions(PartitionPruner.scala:129)
at
org.apache.flink.table.planner.plan.rules.logical.PushPartitionIntoTableSourceScanRule.lambda$onMatch$3(PushPartitionIntoTableSourceScanRule.java:160)
{noformat}
With metrics disabled the same query plans and prunes to the matching partition.
* *Constant folding is skipped.* A call with constant arguments, such as
{{myudf('abc')}}, is no longer folded. The planner logs a WARN with the same
exception on every reduction attempt (6 to 13 times for one small query) and
evaluates the call per row. The reducer gives up on the whole batch of
constants, so other constants in the same projection, for example
{{UPPER('xyz')}}, are not folded either.
* *A {{SpecializedFunction}} evaluator fails the job.* A user function whose
{{specialize}} calls another user-defined function through
{{SpecializedContext#createEvaluator}} fails at runtime with the same
{{IllegalStateException}}, because the generated evaluator is opened without a
runtime context ({{DefaultExpressionEvaluator.java:62}}). With metrics disabled
the job runs.
*How to reproduce*
Reproduced on master in planner tests, using the {{values}} test connector with
{{'partition-list' = 'p:A;p:B'}} for the partitioned table. {{myudf}} is a
deterministic {{ScalarFunction}} that returns {{s.toUpperCase() + "!"}}.
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
-- part_src (a INT, p STRING) PARTITIONED BY (p)
EXPLAIN SELECT a, p FROM part_src WHERE myudf(p) = 'A!'; -- fails, see
above
EXPLAIN SELECT a, p FROM part_src WHERE STARTSWITH(p, 'A'); -- fails the
same way
EXPLAIN SELECT a, p FROM part_src WHERE UPPER(p) = 'A'; -- plans and
prunes (control)
-- src (a INT, b STRING)
EXPLAIN SELECT myudf('abc') AS c, a FROM src; -- not folded,
WARN logged
{code}
With metrics disabled, the first two plan with only partition {{p=A}} scanned,
and {{myudf('abc')}} is folded to {{'ABC!'}}.
*Cause*
With metrics enabled, {{CodeGeneratorContext#addReusableUdfMetrics}} adds
{{UdfMetrics.register(getRuntimeContext().getMetricGroup(), ...)}} to the
generated {{open()}}. {{ExpressionReducer}} ({{ExpressionReducer.scala:111}})
and {{PartitionPruner}} ({{PartitionPruner.scala:129}}) call {{open()}} on
generated functions while planning, and {{DefaultExpressionEvaluator}} opens
its generated evaluator at runtime; none of them has a runtime context. The
reducer catches the exception and leaves the expressions unreduced. The pruner
does not catch it, so planning fails.
*Proposed fix*
Do not instrument calls generated for these paths. The planner paths use a
{{ConstantCodeGeneratorContext}} ({{ExpressionReducer.scala:78}},
{{PartitionPruner.scala:96}}) and the evaluator uses an
{{EvaluatorCodeGeneratorContext}} ({{BridgingFunctionGenUtil.scala:1002}}), so
{{BridgingFunctionGenUtil#udfMetricsTermIfEnabled}} can skip the metrics for
these two contexts. Plan-time evaluation is not part of the running job, so it
should not report job metrics. FLINK-19112 drew the same line for function
metric groups during optimization. Add planner tests for both paths with
metrics enabled.
h3. 2. Built-in functions are metered
*What happens*
With {{table.exec.udf-metric-enabled}} set to true, built-in functions that
have a runtime implementation class get a UDF metric group, as if they were
user-defined functions. This includes the internal functions the planner adds
for {{UNNEST}} and {{INTERSECT ALL}}, so a query that calls no function at all
can get a group such as {{udf.$UNNEST_ROWS$1}}. The metrics docs and the option
description say these metrics are registered for SQL/Table user-defined
functions.
*How to reproduce*
Reproduced on master in a planner integration test with an in-memory metric
reporter, streaming mode, sample interval 1:
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
-- src (id INT, arr ARRAY<INT>, s STRING)
INSERT INTO sink SELECT TYPEOF(id), ARRAY_CONTAINS(arr, id), UPPER(s) FROM src;
INSERT INTO sink2 SELECT id, x FROM src CROSS JOIN UNNEST(arr) AS T(x);
INSERT INTO sink3 SELECT id FROM src INTERSECT ALL SELECT id FROM src;
{code}
Registered groups:
{noformat}
Calc: udf.TYPEOF, udf.ARRAY_CONTAINS (no group for UPPER)
Correlate: udf.$UNNEST_ROWS$1
Correlate: udf.$REPLICATE_ROWS$1
{noformat}
With {{table.exec.udf-metric-enabled}} set to false, no group is registered. A
user-defined function in the same query gets only its own group, as expected.
*Cause*
Built-in functions with a runtime class are generated through
{{BridgingSqlFunctionCallGen}}, the same entry as user-defined functions, and
that entry always passes a metric name
({{BridgingSqlFunctionCallGen.scala:82}}). Built-ins that {{ExprCodeGenerator}}
handles explicitly, such as {{UPPER}}, do not take this path. On release-2.4
about 60 public built-ins are affected (scalar or table built-ins with a
runtime class, for example the {{ARRAY_\*}} and {{MAP_\*}} functions,
{{TYPEOF}} and {{STARTSWITH}}), plus the internal {{$UNNEST_ROWS$1}},
{{$UNNEST_ROWS_WITH_ORDINALITY$1}} and {{$REPLICATE_ROWS$1}}.
Besides the extra groups and the sampling cost, this makes problem 1 reachable
with built-ins alone, for example {{STARTSWITH}} in a partition filter.
*Proposed fix*
Do not pass a metric name in {{BridgingSqlFunctionCallGen}} when the function
definition is a {{BuiltInFunctionDefinition}}. That also covers the internal
UNNEST and REPLICATE_ROWS functions. Add a test that a query using only
built-ins registers no UDF metric group.
h3. 3. A UDF used in several parts of one operator loses samples
*What happens*
With {{table.exec.udf-metric-enabled}} set to true, a user-defined function
that is called both in the arguments of a table function and in the condition
of the same correlate reports only part of its calls. The second registration
of {{udf.<name>}} on the operator's metric group is rejected, and every sample
and exception recorded through it is lost:
{noformat}
Name collision: Group already contains a Metric with the name
'udfProcessingTime'. Metric will not be reported.
Name collision: Group already contains a Metric with the name
'udfExceptionCount'. Metric will not be reported.
{noformat}
*How to reproduce*
Reproduced on master in a planner integration test with an in-memory metric
reporter, sample interval 1, 3 source rows. {{probetableudf}} emits each input
twice.
{code:sql}
SET 'table.exec.udf-metric-enabled' = 'true';
INSERT INTO sink SELECT x FROM src, LATERAL TABLE(probetableudf(probeudf(id)))
AS T(x) WHERE probeudf(x) > 0;
{code}
The plan has one {{Correlate}} with
{{invocation=\[probetableudf(probeudf($cor0.id))\]}} and
{{condition=\[>(probeudf($0), 0)\]}}.
|| Case || Expected {{udf.probeudf}} || Reported ||
| invocation and condition | 9 samples | 6 samples, plus the two WARNs above |
| invocation and condition, the function throws once on the invocation side | 6
samples, 1 exception | 4 samples, 0 exceptions |
| invocation only | 3 samples | 3 samples |
| condition only | 6 samples | 6 samples |
Counters inside the functions confirm that all calls are made. Only the metrics
are lost.
*Cause*
The correlate's condition is generated into a separate collector class with its
own {{CodeGeneratorContext}} ({{CorrelateCodeGenerator.scala:196}}).
{{CodeGeneratorContext#addReusableUdfMetrics}} registers each function name
once per context, so the operator class and the collector class each call
{{UdfMetrics.register}} for the same name on the same operator metric group.
The collector opens first, and the operator's registration is the one rejected.
Other operators that generate several classes on one metric group lose samples
the same way when one function appears in more than one of them. Reproduced on
master with sample interval 1:
|| Operator || Query shape || Calls || Reported samples ||
| Lookup join | {{cudf}} in the pre-filter and in the remaining join condition
| 5 | 2 |
| MATCH_RECOGNIZE | {{cudf}} in {{DEFINE}} and in {{MEASURES}} | 6 | 3 |
Multi-way joins and delta joins generate their conditions the same way (code
reading, not run).
*Proposed fix*
Register each function name once per operator rather than once per generated
class: {{UdfMetrics.register}} returns the existing handle when the same
operator metric group already has one for that name, so every generated class
of the operator records into one histogram and one counter. Add tests for the
correlate and lookup join cases above.
> Refine UDF metrics registration
> -------------------------------
>
> Key: FLINK-41585
> URL: https://issues.apache.org/jira/browse/FLINK-41585
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Table SQL / Runtime
> Affects Versions: 2.4.0
> Reporter: Weiqing Yang
> Assignee: Weiqing Yang
> Priority: Not a Priority
> Fix For: 2.4.0
>
>
> With {{table.exec.udf-metric-enabled}} set to true:
> # *Evaluation outside a running operator fails.* Code generated for constant
> folding, partition pruning, watermark generators pushed into a source and
> {{SpecializedFunction}} expression evaluators registers metrics, but is
> opened without a runtime context. A user-defined function or {{STARTSWITH}}
> in a partition filter fails planning with {{IllegalStateException: The
> runtime context has not been initialized}}. Calls with constant arguments are
> not folded. A user-defined function in a pushed-down {{WATERMARK}}
> expression, or called by a {{SpecializedFunction}} evaluator, fails the job
> at runtime.
> # *Built-in functions get UDF metric groups.* Built-ins with a runtime class,
> such as {{TYPEOF}} and {{ARRAY_CONTAINS}}, and the internal functions used
> for {{UNNEST}} and {{INTERSECT ALL}}, register {{udf.*}} groups.
> # *Calls are not reported when a function is used in several parts of one
> operator.* For example in a correlate's table function arguments and its
> condition, in a lookup join's pre-filter and join condition, or in {{DEFINE}}
> and {{MEASURES}} of {{MATCH_RECOGNIZE}}. Each part registers {{udf.<name>}}
> on the same operator metric group, only the first registration is kept, and a
> {{Name collision}} WARN is logged.
> {code:sql}
> SET 'table.exec.udf-metric-enabled' = 'true';
> -- 1: fails planning (part_src is partitioned by p)
> SELECT a FROM part_src WHERE myudf(p) = 'A';
> -- 2: registers udf.TYPEOF
> SELECT TYPEOF(id) FROM src;
> -- 3: some calls of probeudf are not reported
> SELECT x FROM src, LATERAL TABLE(tf(probeudf(id))) AS T(x) WHERE probeudf(x)
> > 0;
> {code}
> *Proposed fix*
> # Do not register metrics in code generated for constant folding, partition
> pruning, pushed-down watermark generators and expression evaluators.
> # Do not register metrics for built-in functions.
> # Register each function once per operator metric group, and share it across
> the operator's generated classes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)