Arvid Heise created FLINK-40940:
-----------------------------------
Summary: HTTP lookup source drops the row for a failed call with
continue-on-error unless the planner calls applyReadableMetadata
Key: FLINK-40940
URL: https://issues.apache.org/jira/browse/FLINK-40940
Project: Flink
Issue Type: Bug
Components: Connectors / HTTP
Affects Versions: http-1.0.0
Reporter: Arvid Heise
Assignee: Arvid Heise
With \{{http.source.lookup.continue-on-error}} = true, the docs state that a
failed lookup call completes the join "without content in the expected
enrichment columns", i.e. the enrichment columns are null. For a table without
metadata columns this only holds if the planner happened to call
\{{SupportsReadingMetadata#applyReadableMetadata}}:
{code:sql}
CREATE TABLE http_lookup (id STRING, name STRING, age INT) WITH (
'connector' = 'rest-lookup',
'url' = '...',
'format' = 'json',
'http.source.lookup.continue-on-error' = 'true');
-- the endpoint returns an error for the key
SELECT o.id, l.name FROM orders AS o
JOIN http_lookup FOR SYSTEM_TIME AS OF o.proc_time AS l ON o.id = l.id;
{code}
* If a projection is pushed into the lookup source (as in this query), the
planner records an empty ReadingMetadata ability, \{{applyReadableMetadata([],
...)}} sets \{{producedDataType}}, and the failed call yields a row with null
enrichment columns.
* If no projection is pushed (the query uses all columns),
\{{producedDataType}} stays null and \{{HttpTableLookupFunction}} returns no
row ("return the same way as ignore", HttpTableLookupFunction.java:174), so the
inner lookup join drops the row.
{\{HttpLookupTableSource}} sets \{{producedDataType}} only in
\{{applyReadableMetadata}} (HttpLookupTableSource.java:339); the constructor
and \{{applyProjection}} leave it unset, and \{{copy()}} does not carry it or
\{{metadataKeys}} over. The lookup function uses it to size the output row when
the response has no data.
FLINK-38569 stops the planner from recording the empty ReadingMetadata ability,
which would make the second behavior apply to all queries against tables
without metadata columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
