[
https://issues.apache.org/jira/browse/FLINK-39368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-39368:
-----------------------------------
Labels: pull-request-available (was: )
> [connector-http] Add User-Agent configuration support for HTTP sink and lookup
> ------------------------------------------------------------------------------
>
> Key: FLINK-39368
> URL: https://issues.apache.org/jira/browse/FLINK-39368
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / HTTP
> Reporter: featzhang
> Priority: Minor
> Labels: pull-request-available
>
> h2. Summary
> Add a dedicated {{http.user.agent}} configuration option for both the HTTP
> lookup source and HTTP async sink, providing a sensible default value of
> {{flink-http-connector}} so the connector is self-identifying without any
> explicit user configuration.
> h2. Motivation
> The HTTP connector already supports setting arbitrary headers via the prefix
> mechanism:
> * Sink: {{'http.sink.header.User-Agent' = 'my-agent/1.0'}}
> * Lookup: {{'http.source.lookup.header.User-Agent' = 'my-agent/1.0'}}
> This works, but has two shortcomings specific to the {{User-Agent}} header:
> # *No default value.* Without explicit configuration, the connector sends
> requests with no {{User-Agent}} header (or the JDK's default
> {{Java/...}}).
> Many HTTP servers use {{User-Agent}} for logging, rate limiting, or
> routing.
> A meaningful default ({{flink-http-connector}}) makes the connector
> self-identifying out of the box, which the prefix-based mechanism cannot
> provide.
> # *Discoverability.* Prefix-based header keys such as
> {{http.sink.header.User-Agent}} are not enumerated in the connector's
> documented {{ConfigOption}} set and do not appear in tooling such as Flink
> SQL IDE auto-completion. A named option is self-documenting and easier for
> users to find.
> If the community agrees the existing custom-header mechanism is sufficient
> and
> a dedicated option adds unnecessary complexity, we are open to closing this
> issue. The central question is whether a connector-level default value for
> {{User-Agent}} is desirable.
> h2. Proposed Changes
> h3. New configuration options
> || Option key || Component || Default || Description ||
> | {{http.user.agent}} | HTTP Lookup Source | {{flink-http-connector}} |
> User-Agent header value sent with every lookup request |
> | {{http.user.agent}} | HTTP Async Sink | {{flink-http-connector}} |
> User-Agent header value sent with every sink request |
> h3. Code changes
> * Add {{ConfigOption<String> SOURCE_LOOKUP_USER_AGENT}} (key
> {{http.user.agent}},
> default {{flink-http-connector}}) to {{HttpLookupConnectorOptions}}.
> * Add {{ConfigOption<String> USER_AGENT}} (key {{http.user.agent}},
> default {{flink-http-connector}}) to {{HttpDynamicSinkConnectorOptions}}.
> * Register both options in {{optionalOptions()}} of their respective
> factories
> ({{HttpLookupTableSourceFactory}}, {{HttpDynamicTableSinkFactory}}) so
> Flink's
> option validation does not reject the key.
> * In {{RequestFactoryBase}}, read the value via
> {{readableConfig.get(SOURCE_LOOKUP_USER_AGENT)}} and inject it into the
> request header map.
> * In {{JavaNetSinkHttpClient}}, read the value via
> {{properties.getProperty("http.user.agent", "flink-http-connector")}} and
> inject it into the header map.
> h2. Example
> *Before* — user must set the header explicitly or no User-Agent is sent:
> {code:sql}
> CREATE TABLE http_lookup (id BIGINT, name STRING) WITH (
> 'connector' = 'http',
> 'url' = 'http://example.com/api',
> 'format' = 'json',
> 'http.source.lookup.header.User-Agent' = 'my-flink-job/1.0'
> );
> CREATE TABLE http_sink (id BIGINT, name STRING) WITH (
> 'connector' = 'http-async-sink',
> 'url' = 'http://example.com/api',
> 'format' = 'json',
> 'http.sink.header.User-Agent' = 'my-flink-job/1.0'
> );
> {code}
> *After* — connector sends {{User-Agent: flink-http-connector}} by default;
> users can override with a single short key:
> {code:sql}
> CREATE TABLE http_lookup (id BIGINT, name STRING) WITH (
> 'connector' = 'http',
> 'url' = 'http://example.com/api',
> 'format' = 'json',
> 'http.user.agent' = 'my-flink-job/1.0' -- optional, defaults to
> flink-http-connector
> );
> CREATE TABLE http_sink (id BIGINT, name STRING) WITH (
> 'connector' = 'http-async-sink',
> 'url' = 'http://example.com/api',
> 'format' = 'json',
> 'http.user.agent' = 'my-flink-job/1.0' -- optional, defaults to
> flink-http-connector
> );
> {code}
> h2. Testing
> * {{JavaNetSinkHttpClientTest}}: updated existing header count assertions;
> added {{shouldBuildClientWithUserAgentHeader()}} and
> {{shouldUseDefaultUserAgentWhenNotConfigured()}}.
> * {{JavaNetHttpPollingClientTest}}: same additions for the lookup client
> side.
> * All 26 unit tests pass.
> h2. Related
> * PR: https://github.com/apache/flink-connector-http/pull/34
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
