[
https://issues.apache.org/jira/browse/FLINK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiqing Yang updated FLINK-41585:
---------------------------------
Summary: Refine UDF metrics registration (was: UDF metrics: planning fails
for UDFs in partition filters, built-in functions are metered, and operators
lose samples)
> Refine UDF metrics registration
> -------------------------------
>
> Key: FLINK-41585
> URL: https://issues.apache.org/jira/browse/FLINK-41585
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Table SQL / Runtime
> Affects Versions: 2.4.0
> Reporter: Weiqing Yang
> Assignee: Weiqing Yang
> Priority: Not a Priority
> Fix For: 2.4.0
>
>
> Release testing of FLIP-485 (FLINK-41450) found three problems with
> {{table.exec.udf-metric-enabled}} set to true. They are all in UDF metrics
> registration, so they are fixed together in one pull request.
> # Code evaluated outside a running operator calls {{getRuntimeContext()}}:
> planning fails when a UDF or a built-in such as {{STARTSWITH}} is in a
> partition filter, constant calls are no longer folded, and a
> {{SpecializedFunction}} evaluator fails the job.
> # Built-in functions such as {{TYPEOF}} and {{ARRAY_CONTAINS}}, and the
> internal UNNEST function, get UDF metric groups.
> # A UDF used in more than one part of one operator (correlate, lookup join,
> MATCH_RECOGNIZE) loses the samples of all but one part.
> h3. 1. Code evaluated outside a running operator fails
> *What happens*
> With {{table.exec.udf-metric-enabled}} set to true, three kinds of evaluation
> outside a running operator break:
> * *Partition pruning fails the query.* A filter on a partition column that
> calls a user-defined function, or a built-in such as {{STARTSWITH}}, fails at
> planning, in streaming and batch:
> {noformat}
> java.lang.IllegalStateException: The runtime context has not been initialized.
> at
> org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:51)
> at PartitionPruner$13.open(Unknown Source)
> at
> org.apache.flink.table.planner.plan.utils.PartitionPruner$.prunePartitions(PartitionPruner.scala:129)
> at
> org.apache.flink.table.planner.plan.rules.logical.PushPartitionIntoTableSourceScanRule.lambda$onMatch$3(PushPartitionIntoTableSourceScanRule.java:160)
> {noformat}
> With metrics disabled the same query plans and prunes to the matching
> partition.
> * *Constant folding is skipped.* A call with constant arguments, such as
> {{myudf('abc')}}, is no longer folded. The planner logs a WARN with the same
> exception on every reduction attempt (6 to 13 times for one small query) and
> evaluates the call per row. The reducer gives up on the whole batch of
> constants, so other constants in the same projection, for example
> {{UPPER('xyz')}}, are not folded either.
> * *A {{SpecializedFunction}} evaluator fails the job.* A user function whose
> {{specialize}} calls another user-defined function through
> {{SpecializedContext#createEvaluator}} fails at runtime with the same
> {{IllegalStateException}}, because the generated evaluator is opened without
> a runtime context ({{DefaultExpressionEvaluator.java:62}}). With metrics
> disabled the job runs.
> *How to reproduce*
> Reproduced on master in planner tests, using the {{values}} test connector
> with {{'partition-list' = 'p:A;p:B'}} for the partitioned table. {{myudf}} is
> a deterministic {{ScalarFunction}} that returns {{s.toUpperCase() + "!"}}.
> {code:sql}
> SET 'table.exec.udf-metric-enabled' = 'true';
> -- part_src (a INT, p STRING) PARTITIONED BY (p)
> EXPLAIN SELECT a, p FROM part_src WHERE myudf(p) = 'A!'; -- fails, see
> above
> EXPLAIN SELECT a, p FROM part_src WHERE STARTSWITH(p, 'A'); -- fails the
> same way
> EXPLAIN SELECT a, p FROM part_src WHERE UPPER(p) = 'A'; -- plans and
> prunes (control)
> -- src (a INT, b STRING)
> EXPLAIN SELECT myudf('abc') AS c, a FROM src; -- not
> folded, WARN logged
> {code}
> With metrics disabled, the first two plan with only partition {{p=A}}
> scanned, and {{myudf('abc')}} is folded to {{'ABC!'}}.
> *Cause*
> With metrics enabled, {{CodeGeneratorContext#addReusableUdfMetrics}} adds
> {{UdfMetrics.register(getRuntimeContext().getMetricGroup(), ...)}} to the
> generated {{open()}}. {{ExpressionReducer}} ({{ExpressionReducer.scala:111}})
> and {{PartitionPruner}} ({{PartitionPruner.scala:129}}) call {{open()}} on
> generated functions while planning, and {{DefaultExpressionEvaluator}} opens
> its generated evaluator at runtime; none of them has a runtime context. The
> reducer catches the exception and leaves the expressions unreduced. The
> pruner does not catch it, so planning fails.
> *Proposed fix*
> Do not instrument calls generated for these paths. The planner paths use a
> {{ConstantCodeGeneratorContext}} ({{ExpressionReducer.scala:78}},
> {{PartitionPruner.scala:96}}) and the evaluator uses an
> {{EvaluatorCodeGeneratorContext}} ({{BridgingFunctionGenUtil.scala:1002}}),
> so {{BridgingFunctionGenUtil#udfMetricsTermIfEnabled}} can skip the metrics
> for these two contexts. Plan-time evaluation is not part of the running job,
> so it should not report job metrics. FLINK-19112 drew the same line for
> function metric groups during optimization. Add planner tests for both paths
> with metrics enabled.
> h3. 2. Built-in functions are metered
> *What happens*
> With {{table.exec.udf-metric-enabled}} set to true, built-in functions that
> have a runtime implementation class get a UDF metric group, as if they were
> user-defined functions. This includes the internal functions the planner adds
> for {{UNNEST}} and {{INTERSECT ALL}}, so a query that calls no function at
> all can get a group such as {{udf.$UNNEST_ROWS$1}}. The metrics docs and the
> option description say these metrics are registered for SQL/Table
> user-defined functions.
> *How to reproduce*
> Reproduced on master in a planner integration test with an in-memory metric
> reporter, streaming mode, sample interval 1:
> {code:sql}
> SET 'table.exec.udf-metric-enabled' = 'true';
> -- src (id INT, arr ARRAY<INT>, s STRING)
> INSERT INTO sink SELECT TYPEOF(id), ARRAY_CONTAINS(arr, id), UPPER(s) FROM
> src;
> INSERT INTO sink2 SELECT id, x FROM src CROSS JOIN UNNEST(arr) AS T(x);
> INSERT INTO sink3 SELECT id FROM src INTERSECT ALL SELECT id FROM src;
> {code}
> Registered groups:
> {noformat}
> Calc: udf.TYPEOF, udf.ARRAY_CONTAINS (no group for UPPER)
> Correlate: udf.$UNNEST_ROWS$1
> Correlate: udf.$REPLICATE_ROWS$1
> {noformat}
> With {{table.exec.udf-metric-enabled}} set to false, no group is registered.
> A user-defined function in the same query gets only its own group, as
> expected.
> *Cause*
> Built-in functions with a runtime class are generated through
> {{BridgingSqlFunctionCallGen}}, the same entry as user-defined functions, and
> that entry always passes a metric name
> ({{BridgingSqlFunctionCallGen.scala:82}}). Built-ins that
> {{ExprCodeGenerator}} handles explicitly, such as {{UPPER}}, do not take this
> path. On release-2.4 about 60 public built-ins are affected (scalar or table
> built-ins with a runtime class, for example the {{ARRAY_\*}} and {{MAP_\*}}
> functions, {{TYPEOF}} and {{STARTSWITH}}), plus the internal
> {{$UNNEST_ROWS$1}}, {{$UNNEST_ROWS_WITH_ORDINALITY$1}} and
> {{$REPLICATE_ROWS$1}}.
> Besides the extra groups and the sampling cost, this makes problem 1
> reachable with built-ins alone, for example {{STARTSWITH}} in a partition
> filter.
> *Proposed fix*
> Do not pass a metric name in {{BridgingSqlFunctionCallGen}} when the function
> definition is a {{BuiltInFunctionDefinition}}. That also covers the internal
> UNNEST and REPLICATE_ROWS functions. Add a test that a query using only
> built-ins registers no UDF metric group.
> h3. 3. A UDF used in several parts of one operator loses samples
> *What happens*
> With {{table.exec.udf-metric-enabled}} set to true, a user-defined function
> that is called both in the arguments of a table function and in the condition
> of the same correlate reports only part of its calls. The second registration
> of {{udf.<name>}} on the operator's metric group is rejected, and every
> sample and exception recorded through it is lost:
> {noformat}
> Name collision: Group already contains a Metric with the name
> 'udfProcessingTime'. Metric will not be reported.
> Name collision: Group already contains a Metric with the name
> 'udfExceptionCount'. Metric will not be reported.
> {noformat}
> *How to reproduce*
> Reproduced on master in a planner integration test with an in-memory metric
> reporter, sample interval 1, 3 source rows. {{probetableudf}} emits each
> input twice.
> {code:sql}
> SET 'table.exec.udf-metric-enabled' = 'true';
> INSERT INTO sink SELECT x FROM src, LATERAL
> TABLE(probetableudf(probeudf(id))) AS T(x) WHERE probeudf(x) > 0;
> {code}
> The plan has one {{Correlate}} with
> {{invocation=\[probetableudf(probeudf($cor0.id))\]}} and
> {{condition=\[>(probeudf($0), 0)\]}}.
> || Case || Expected {{udf.probeudf}} || Reported ||
> | invocation and condition | 9 samples | 6 samples, plus the two WARNs above |
> | invocation and condition, the function throws once on the invocation side |
> 6 samples, 1 exception | 4 samples, 0 exceptions |
> | invocation only | 3 samples | 3 samples |
> | condition only | 6 samples | 6 samples |
> Counters inside the functions confirm that all calls are made. Only the
> metrics are lost.
> *Cause*
> The correlate's condition is generated into a separate collector class with
> its own {{CodeGeneratorContext}} ({{CorrelateCodeGenerator.scala:196}}).
> {{CodeGeneratorContext#addReusableUdfMetrics}} registers each function name
> once per context, so the operator class and the collector class each call
> {{UdfMetrics.register}} for the same name on the same operator metric group.
> The collector opens first, and the operator's registration is the one
> rejected.
> Other operators that generate several classes on one metric group lose
> samples the same way when one function appears in more than one of them.
> Reproduced on master with sample interval 1:
> || Operator || Query shape || Calls || Reported samples ||
> | Lookup join | {{cudf}} in the pre-filter and in the remaining join
> condition | 5 | 2 |
> | MATCH_RECOGNIZE | {{cudf}} in {{DEFINE}} and in {{MEASURES}} | 6 | 3 |
> Multi-way joins and delta joins generate their conditions the same way (code
> reading, not run).
> *Proposed fix*
> Register each function name once per operator rather than once per generated
> class: {{UdfMetrics.register}} returns the existing handle when the same
> operator metric group already has one for that name, so every generated class
> of the operator records into one histogram and one counter. Add tests for the
> correlate and lookup join cases above.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)