AHeise opened a new pull request, #56: URL: https://github.com/apache/flink-connector-http/pull/56
## What is the purpose of the change With `http.source.lookup.continue-on-error` = `true`, the docs say a failed lookup call completes the join with null enrichment columns. `HttpTableLookupFunction` only does this when `producedDataType` is set, and `HttpLookupTableSource` sets it only in `applyReadableMetadata`. For a table without metadata columns, the planner calls `applyReadableMetadata` only when it pushes a projection, so: - without a pushed projection, a failed call returns no row and the inner lookup join drops the probe row; - with a pushed projection, an ignored status code (`http.source.lookup.ignored-response-codes`) emits an all-null row, although the docs say no row is emitted when there are no metadata columns. FLINK-38569 stops the planner from recording the empty `ReadingMetadata` ability, which would turn the first behavior into the default for every table without metadata columns. ## Brief change log Three commits: 1. **Populate join keys on lookup rows without response data**: a row emitted without response data (to carry metadata or, after this PR, for a failed call) fills the lookup-side join key columns from the probe key, as rows built from response data already do. The diff is mostly re-indentation of the existing join key block (view with whitespace ignored). 2. **Retain readable metadata in `HttpLookupTableSource#copy`**: `copy()` dropped `metadataKeys` and `producedDataType`. 3. **Emit null enrichment row for failed lookup calls**: emission is decided by `HttpCompletionState`. Failed calls (`HTTP_ERROR_STATUS`, `EXCEPTION`, `UNABLE_TO_DESERIALIZE_RESPONSE`) always yield a row with null enrichment columns. Ignored or empty responses yield a row only when metadata columns are declared. The physical arity comes from the physical row type, and the unused `producedDataType` is removed from the source and the lookup function. Behavior change: an ignored status code or an empty response, on a table without metadata columns and with a pushed projection, now yields no row (the documented behavior), where before it yielded an all-null row. Not addressed (pre-existing): NOT NULL enrichment columns of such rows are null, while the docs mention default values for non-nullable columns. ## Verifying this change - `HttpTableLookupFunctionTest` (new): for every completion state, a response without data emits the expected row (or none), with and without metadata columns, including the join key. - `HttpLookupTableSourceTest#copyRetainsReadableMetadata`: `copy()` keeps metadata keys. - `HttpLookupTableSourceITCaseTest#testHttpLookupJoinParameterized`: new bad-status specs without metadata columns for GET/POST/PUT, sync/async and continue-on-error on/off. With continue-on-error, all four probe rows come through with null enrichment. These cases fail without commit 3. - `mvn verify -pl flink-connector-http` passes on each commit (JDK 17, Flink 2.2.0). ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** (the `HttpTableLookupFunction` constructor loses its `producedDataType` parameter; the class is not annotated) - The serializers: **no** - The runtime per-record code paths (performance sensitive): **yes** (lookup function result assembly; no extra work per record) - Anything that affects deployment or recovery: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? not applicable (the change aligns the behavior with the existing docs) ## Was generative AI tooling used to co-author this PR? - [x] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
