[
https://issues.apache.org/jira/browse/FLINK-39368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
featzhang updated FLINK-39368:
------------------------------
Description:
h2. Summary
Add a dedicated {{http.user.agent}} configuration option for both the HTTP
lookup source and HTTP async sink, providing a sensible default value of
{{flink-http-connector}} so the connector is self-identifying without any
explicit user configuration.
h2. Motivation
The HTTP connector already supports setting arbitrary headers via the prefix
mechanism:
* Sink: {{'http.sink.header.User-Agent' = 'my-agent/1.0'}}
* Lookup: {{'http.source.lookup.header.User-Agent' = 'my-agent/1.0'}}
This works, but has two shortcomings specific to the {{User-Agent}} header:
# *No default value.* Without explicit configuration, the connector sends
requests with no {{User-Agent}} header (or the JDK's default {{Java/...}}).
Many HTTP servers use {{User-Agent}} for logging, rate limiting, or routing.
A meaningful default ({{flink-http-connector}}) makes the connector
self-identifying out of the box, which the prefix-based mechanism cannot
provide.
# *Discoverability.* Prefix-based header keys such as
{{http.sink.header.User-Agent}} are not enumerated in the connector's
documented {{ConfigOption}} set and do not appear in tooling such as Flink
SQL IDE auto-completion. A named option is self-documenting and easier for
users to find.
If the community agrees the existing custom-header mechanism is sufficient and
a dedicated option adds unnecessary complexity, we are open to closing this
issue. The central question is whether a connector-level default value for
{{User-Agent}} is desirable.
h2. Proposed Changes
h3. New configuration options
|| Option key || Component || Default || Description ||
| {{http.user.agent}} | HTTP Lookup Source | {{flink-http-connector}} |
User-Agent header value sent with every lookup request |
| {{http.user.agent}} | HTTP Async Sink | {{flink-http-connector}} |
User-Agent header value sent with every sink request |
h3. Code changes
* Add {{ConfigOption<String> SOURCE_LOOKUP_USER_AGENT}} (key
{{http.user.agent}},
default {{flink-http-connector}}) to {{HttpLookupConnectorOptions}}.
* Add {{ConfigOption<String> USER_AGENT}} (key {{http.user.agent}},
default {{flink-http-connector}}) to {{HttpDynamicSinkConnectorOptions}}.
* Register both options in {{optionalOptions()}} of their respective factories
({{HttpLookupTableSourceFactory}}, {{HttpDynamicTableSinkFactory}}) so
Flink's
option validation does not reject the key.
* In {{RequestFactoryBase}}, read the value via
{{readableConfig.get(SOURCE_LOOKUP_USER_AGENT)}} and inject it into the
request header map.
* In {{JavaNetSinkHttpClient}}, read the value via
{{properties.getProperty("http.user.agent", "flink-http-connector")}} and
inject it into the header map.
h2. Example
*Before* — user must set the header explicitly or no User-Agent is sent:
{code:sql}
CREATE TABLE http_lookup (id BIGINT, name STRING) WITH (
'connector' = 'http',
'url' = 'http://example.com/api',
'format' = 'json',
'http.source.lookup.header.User-Agent' = 'my-flink-job/1.0'
);
CREATE TABLE http_sink (id BIGINT, name STRING) WITH (
'connector' = 'http-async-sink',
'url' = 'http://example.com/api',
'format' = 'json',
'http.sink.header.User-Agent' = 'my-flink-job/1.0'
);
{code}
*After* — connector sends {{User-Agent: flink-http-connector}} by default;
users can override with a single short key:
{code:sql}
CREATE TABLE http_lookup (id BIGINT, name STRING) WITH (
'connector' = 'http',
'url' = 'http://example.com/api',
'format' = 'json',
'http.user.agent' = 'my-flink-job/1.0' -- optional, defaults to
flink-http-connector
);
CREATE TABLE http_sink (id BIGINT, name STRING) WITH (
'connector' = 'http-async-sink',
'url' = 'http://example.com/api',
'format' = 'json',
'http.user.agent' = 'my-flink-job/1.0' -- optional, defaults to
flink-http-connector
);
{code}
h2. Testing
* {{JavaNetSinkHttpClientTest}}: updated existing header count assertions;
added {{shouldBuildClientWithUserAgentHeader()}} and
{{shouldUseDefaultUserAgentWhenNotConfigured()}}.
* {{JavaNetHttpPollingClientTest}}: same additions for the lookup client side.
* All 26 unit tests pass.
h2. Related
* PR: https://github.com/apache/flink-connector-http/pull/34
was:
h2. Summary
Currently, the HTTP connector does not set a {{User-Agent}} header by default,
and there is no dedicated configuration option to control it. Users who need
to identify their Flink jobs to downstream HTTP servers must resort to
manually
configuring a raw header property, which is not discoverable and lacks a
sensible default.
This issue tracks adding first-class {{User-Agent}} configuration support to
both the HTTP sink and HTTP lookup source.
h2. Motivation
Many HTTP servers use the {{User-Agent}} header for:
- Request logging and traffic attribution
- Rate limiting policies per client
- Conditional routing or access control
A dedicated option with a meaningful default ({{flink-http-connector}}) makes
the connector self-identifying out of the box, while still allowing users to
override it per job.
h2. Proposed Changes
h3. New configuration options
|| Option key || Component || Default value || Description ||
| {{http.user.agent}} | HTTP Lookup Source | {{flink-http-connector}} |
User-Agent header value for lookup requests |
| {{http.user.agent}} | HTTP Async Sink | {{flink-http-connector}} |
User-Agent header value for sink requests |
h3. Implementation
* Add {{ConfigOption<String> SOURCE_LOOKUP_USER_AGENT}} to
{{HttpLookupConnectorOptions}} with key {{http.user.agent}} and default
{{flink-http-connector}}.
* Add {{ConfigOption<String> USER_AGENT}} to
{{HttpDynamicSinkConnectorOptions}}
with key {{http.user.agent}} and default {{flink-http-connector}}.
* Register both options in their respective factory {{optionalOptions()}}
methods
({{HttpLookupTableSourceFactory}} and {{HttpDynamicTableSinkFactory}}).
* Apply the configured value in {{RequestFactoryBase}} (lookup) and
{{JavaNetSinkHttpClient}} (sink) when building request headers.
h3. Example usage (Flink SQL)
{code:sql}
-- Lookup source with custom User-Agent
CREATE TABLE http_lookup (
id BIGINT,
name STRING
) WITH (
'connector' = 'http',
'url' = 'http://example.com/api',
'format' = 'json',
'http.user.agent' = 'my-flink-job/1.0'
);
-- Async sink with custom User-Agent
CREATE TABLE http_sink (
id BIGINT,
name STRING
) WITH (
'connector' = 'http-async-sink',
'url' = 'http://example.com/api',
'format' = 'json',
'http.user.agent' = 'my-flink-job/1.0'
);
{code}
h2. Testing
* {{JavaNetSinkHttpClientTest}}: updated existing tests to assert default
{{User-Agent}} header; added {{shouldBuildClientWithUserAgentHeader()}} and
{{shouldUseDefaultUserAgentWhenNotConfigured()}}.
* {{JavaNetHttpPollingClientTest}}: same additions for the lookup client side.
h2. Related
* PR: https://github.com/apache/flink-connector-http/pull/34
> [connector-http] Add User-Agent configuration support for HTTP sink and lookup
> ------------------------------------------------------------------------------
>
> Key: FLINK-39368
> URL: https://issues.apache.org/jira/browse/FLINK-39368
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / HTTP
> Reporter: featzhang
> Priority: Minor
>
> h2. Summary
> Add a dedicated {{http.user.agent}} configuration option for both the HTTP
> lookup source and HTTP async sink, providing a sensible default value of
> {{flink-http-connector}} so the connector is self-identifying without any
> explicit user configuration.
> h2. Motivation
> The HTTP connector already supports setting arbitrary headers via the prefix
> mechanism:
> * Sink: {{'http.sink.header.User-Agent' = 'my-agent/1.0'}}
> * Lookup: {{'http.source.lookup.header.User-Agent' = 'my-agent/1.0'}}
> This works, but has two shortcomings specific to the {{User-Agent}} header:
> # *No default value.* Without explicit configuration, the connector sends
> requests with no {{User-Agent}} header (or the JDK's default
> {{Java/...}}).
> Many HTTP servers use {{User-Agent}} for logging, rate limiting, or
> routing.
> A meaningful default ({{flink-http-connector}}) makes the connector
> self-identifying out of the box, which the prefix-based mechanism cannot
> provide.
> # *Discoverability.* Prefix-based header keys such as
> {{http.sink.header.User-Agent}} are not enumerated in the connector's
> documented {{ConfigOption}} set and do not appear in tooling such as Flink
> SQL IDE auto-completion. A named option is self-documenting and easier for
> users to find.
> If the community agrees the existing custom-header mechanism is sufficient
> and
> a dedicated option adds unnecessary complexity, we are open to closing this
> issue. The central question is whether a connector-level default value for
> {{User-Agent}} is desirable.
> h2. Proposed Changes
> h3. New configuration options
> || Option key || Component || Default || Description ||
> | {{http.user.agent}} | HTTP Lookup Source | {{flink-http-connector}} |
> User-Agent header value sent with every lookup request |
> | {{http.user.agent}} | HTTP Async Sink | {{flink-http-connector}} |
> User-Agent header value sent with every sink request |
> h3. Code changes
> * Add {{ConfigOption<String> SOURCE_LOOKUP_USER_AGENT}} (key
> {{http.user.agent}},
> default {{flink-http-connector}}) to {{HttpLookupConnectorOptions}}.
> * Add {{ConfigOption<String> USER_AGENT}} (key {{http.user.agent}},
> default {{flink-http-connector}}) to {{HttpDynamicSinkConnectorOptions}}.
> * Register both options in {{optionalOptions()}} of their respective
> factories
> ({{HttpLookupTableSourceFactory}}, {{HttpDynamicTableSinkFactory}}) so
> Flink's
> option validation does not reject the key.
> * In {{RequestFactoryBase}}, read the value via
> {{readableConfig.get(SOURCE_LOOKUP_USER_AGENT)}} and inject it into the
> request header map.
> * In {{JavaNetSinkHttpClient}}, read the value via
> {{properties.getProperty("http.user.agent", "flink-http-connector")}} and
> inject it into the header map.
> h2. Example
> *Before* — user must set the header explicitly or no User-Agent is sent:
> {code:sql}
> CREATE TABLE http_lookup (id BIGINT, name STRING) WITH (
> 'connector' = 'http',
> 'url' = 'http://example.com/api',
> 'format' = 'json',
> 'http.source.lookup.header.User-Agent' = 'my-flink-job/1.0'
> );
> CREATE TABLE http_sink (id BIGINT, name STRING) WITH (
> 'connector' = 'http-async-sink',
> 'url' = 'http://example.com/api',
> 'format' = 'json',
> 'http.sink.header.User-Agent' = 'my-flink-job/1.0'
> );
> {code}
> *After* — connector sends {{User-Agent: flink-http-connector}} by default;
> users can override with a single short key:
> {code:sql}
> CREATE TABLE http_lookup (id BIGINT, name STRING) WITH (
> 'connector' = 'http',
> 'url' = 'http://example.com/api',
> 'format' = 'json',
> 'http.user.agent' = 'my-flink-job/1.0' -- optional, defaults to
> flink-http-connector
> );
> CREATE TABLE http_sink (id BIGINT, name STRING) WITH (
> 'connector' = 'http-async-sink',
> 'url' = 'http://example.com/api',
> 'format' = 'json',
> 'http.user.agent' = 'my-flink-job/1.0' -- optional, defaults to
> flink-http-connector
> );
> {code}
> h2. Testing
> * {{JavaNetSinkHttpClientTest}}: updated existing header count assertions;
> added {{shouldBuildClientWithUserAgentHeader()}} and
> {{shouldUseDefaultUserAgentWhenNotConfigured()}}.
> * {{JavaNetHttpPollingClientTest}}: same additions for the lookup client
> side.
> * All 26 unit tests pass.
> h2. Related
> * PR: https://github.com/apache/flink-connector-http/pull/34
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
