[
https://issues.apache.org/jira/browse/FLINK-40940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40940:
-----------------------------------
Labels: pull-request-available (was: )
> HTTP lookup source drops the row for a failed call with continue-on-error
> unless the planner calls applyReadableMetadata
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40940
> URL: https://issues.apache.org/jira/browse/FLINK-40940
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HTTP
> Affects Versions: http-1.0.0
> Reporter: Arvid Heise
> Assignee: Arvid Heise
> Priority: Major
> Labels: pull-request-available
>
> With \{{http.source.lookup.continue-on-error}} = true, the docs state that a
> failed lookup call completes the join "without content in the expected
> enrichment columns", i.e. the enrichment columns are null. For a table
> without metadata columns this only holds if the planner happened to call
> \{{SupportsReadingMetadata#applyReadableMetadata}}:
> {code:sql}
> CREATE TABLE http_lookup (id STRING, name STRING, age INT) WITH (
> 'connector' = 'rest-lookup',
> 'url' = '...',
> 'format' = 'json',
> 'http.source.lookup.continue-on-error' = 'true');
> -- the endpoint returns an error for the key
> SELECT o.id, l.name FROM orders AS o
> JOIN http_lookup FOR SYSTEM_TIME AS OF o.proc_time AS l ON o.id = l.id;
> {code}
> * If a projection is pushed into the lookup source (as in this query), the
> planner records an empty ReadingMetadata ability,
> \{{applyReadableMetadata([], ...)}} sets \{{producedDataType}}, and the
> failed call yields a row with null enrichment columns.
> * If no projection is pushed (the query uses all columns),
> \{{producedDataType}} stays null and \{{HttpTableLookupFunction}} returns no
> row ("return the same way as ignore", HttpTableLookupFunction.java:174), so
> the inner lookup join drops the row.
> {\{HttpLookupTableSource}} sets \{{producedDataType}} only in
> \{{applyReadableMetadata}} (HttpLookupTableSource.java:339); the constructor
> and \{{applyProjection}} leave it unset, and \{{copy()}} does not carry it or
> \{{metadataKeys}} over. The lookup function uses it to size the output row
> when the response has no data.
> FLINK-38569 stops the planner from recording the empty ReadingMetadata
> ability, which would make the second behavior apply to all queries against
> tables without metadata columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
