This is an automated email from the ASF dual-hosted git repository. fcsaky pushed a commit to branch fix-history in repository https://gitbox.apache.org/repos/asf/flink-connector-http.git
commit 448d1a790d641e9e165fd88a8f286ce8c3eb73e5 Author: Harshit Gupta <[email protected]> AuthorDate: Sat Feb 28 14:12:26 2026 +0100 [hotfix] Doc typo fix --- docs/content.zh/docs/connectors/datastream/http.md | 8 +- docs/content.zh/docs/connectors/table/http.md | 100 ++++++++++----------- docs/content/docs/connectors/datastream/http.md | 8 +- docs/content/docs/connectors/table/http.md | 100 ++++++++++----------- 4 files changed, 108 insertions(+), 108 deletions(-) diff --git a/docs/content.zh/docs/connectors/datastream/http.md b/docs/content.zh/docs/connectors/datastream/http.md index c1e7499..f56c848 100644 --- a/docs/content.zh/docs/connectors/datastream/http.md +++ b/docs/content.zh/docs/connectors/datastream/http.md @@ -120,7 +120,7 @@ HttpSink.<String>builder() or ```java -Properties properties = Properties(); +Properties properties = new Properties(); properties.setProperty("http.sink.header.X-Content-Type-Options", "nosniff"); HttpSink.<String>builder() @@ -146,7 +146,7 @@ used - allowing use of globally recognized CAs without the need for configuratio You can also configure the connector to use mTLS. For this simply use `flink.connector.http.security.cert.client` and `flink.connector.http.security.key.client` connector properties to specify paths to the certificate and -private key. The key MUST be in `PCKS8` format. Both PEM and DER keys are +private key. The key MUST be in `PKCS8` format. Both PEM and DER keys are allowed. All properties can be set via Sink's builder `.setProperty(...)` method or through Sink and Source table DDL. @@ -162,7 +162,7 @@ If the used value starts with the prefix `Basic`, or `flink.connector.http.sourc is set to `'true'`, it will be used as header value as is, without any extra modification. ## OIDC Bearer Authentication -The connector supports Bearer Authentication using a HTTP `Authorization` header. The [OAuth 2.0 rcf](https://datatracker.ietf.org/doc/html/rfc6749) mentions [Obtaining Authorization](https://datatracker.ietf.org/doc/html/rfc6749#section-4) +The connector supports Bearer Authentication using a HTTP `Authorization` header. The [OAuth 2.0 RFC](https://datatracker.ietf.org/doc/html/rfc6749) mentions [Obtaining Authorization](https://datatracker.ietf.org/doc/html/rfc6749#section-4) and an authorization grant. OIDC makes use of this [authorisation grant](https://datatracker.ietf.org/doc/html/rfc6749#section-1.3) in a [Token Request](https://openid.net/specs/openid-connect-core-1_0.html#TokenRequest) by including a [OAuth grant type](https://oauth.net/2/grant-types/) and associated properties, the response is the [token response](https://openid.net/specs/openid-connect-core-1_0.html#TokenResponse). If you want to use this authorization then you should supply the `Token Request` body in `application/x-www-form-urlencoded` encoding @@ -175,5 +175,5 @@ be requested if the current time is later than the cached token expiry time minu ### Restrictions at this time * No authentication is applied to the token request. -* The processing does not use the refresh token if it present. +* The processing does not use the refresh token if it is present. {{< top >}} diff --git a/docs/content.zh/docs/connectors/table/http.md b/docs/content.zh/docs/connectors/table/http.md index 2bc8643..5807971 100644 --- a/docs/content.zh/docs/connectors/table/http.md +++ b/docs/content.zh/docs/connectors/table/http.md @@ -163,56 +163,56 @@ Or for REST POST method they will be converted to Json and used as request body. Note the options with the prefix _http_ are the HTTP connector specific options, the others are Flink options. -| Option | Required | Description/Value [...] -|:-----------------------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...] -| connector | required | The Value should be set to _http_ [...] -| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. [...] -| url | required | The base URL that should be use for GET requests. For example _http://localhost:8080/client_ [...] -| asyncPolling | optional | true/false - determines whether Async Polling should be used. Mechanism is based on Flink's Async I/O. [...] -| lookup-method | optional | GET/POST/PUT (and any other) - determines what REST method should be used for lookup REST query. If not specified, `GET` method will be used. [...] -| lookup.cache | optional | Enum possible values: `NONE`, `PARTIAL`. The cache strategy for the lookup table. Currently supports `NONE` (no caching) and `PARTIAL` (caching entries on lookup operation in external API). [...] -| lookup.partial-cache.max-rows | optional | The max number of rows of lookup cache, over this value, the oldest rows will be expired. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] -| lookup.partial-cache.expire-after-write | optional | The max time to live for each rows in lookup cache after writing into the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] -| lookup.partial-cache.expire-after-access | optional | The max time to live for each rows in lookup cache after accessing the entry in the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] -| lookup.partial-cache.cache-missing-key | optional | This is a boolean that defaults to true. Whether to store an empty value into the cache if the lookup key doesn't match any rows in the table. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] -| lookup.max-retries | optional | The max retry times if the lookup failed; default is 3. See the following <a href="#lookup-cache">Lookup Cache</a> section for more detail. Set value 0 to disable retries. [...] -| http.security.cert.server | optional | Comma separated paths to trusted HTTP server certificates that should be added to the connectors trust store. [...] -| http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. [...] -| http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. [...] -| http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. [...] -| http.security.oidc.token.request | optional | OIDC `Token Request` body in `application/x-www-form-urlencoded` encoding [...] -| http.security.oidc.token.endpoint.url | optional | OIDC `Token Endpoint` url, to which the token request will be issued [...] -| http.security.oidc.token.expiry.reduction | optional | OIDC tokens will be requested if the current time is later than the cached token expiry time minus this value. [...] -| http.source.lookup.continue-on-error | optional | When true, the flow will continue on errors, returning row content. When false (the default) the job ends on errors. [...] -| http.source.lookup.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. [...] -| http.source.lookup.request.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 8 threads will be used. [...] -| http.source.lookup.response.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup response processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 4 threads will be used. [...] -| http.source.lookup.use-raw-authorization-header | optional | If set to `'true'`, uses the raw value set for the `Authorization` header, without transformation for Basic Authentication (base64, addition of "Basic " prefix). If not specified, defaults to `'false'`. [...] -| http.source.lookup.request-callback | optional | Specify which `HttpLookupPostRequestCallback` implementation to use. By default, it is set to `slf4j-lookup-logger` corresponding to `Slf4jHttpLookupPostRequestCallback`. [...] -| http.source.lookup.connection.timeout | optional | Source table connection timeout. Default - no value. [...] -| http.source.lookup.http-version | optional | Version of HTTP to use for lookup http requests. The valid values are HTTP_1_1 and HTTP_2, which specify HTTP 1.1 or 2 respectively. This option may be required as HTTP_1_1, if the endpoint is HTTP 1.1, because some http 1.1 endpoints reject HTTP Version 2 calls, with 'Invalid HTTP request received' and 'HTTP/2 upgrade not supported'. [...] -| http.source.lookup.success-codes | optional | Comma separated http codes considered as success response. Use [1-5]XX for groups and '!' character for excluding. The default is 2XX. [...] -| http.source.lookup.retry-codes | optional | Comma separated http codes considered as transient errors. Use [1-5]XX for groups and '!' character for excluding. The default is 500,503,504. [...] -| http.source.lookup.ignored-response-codes | optional | Comma separated http codes. Content for these responses will be ignored. Use [1-5]XX for groups and '!' character for excluding. Ignored responses together with `http.source.lookup.success-codes` are considered as successful. [...] -| http.source.lookup.retry-strategy.type | optional | Auto retry strategy type: fixed-delay (default) or exponential-delay. [...] -| http.source.lookup.retry-strategy.fixed-delay.delay | optional | Fixed-delay interval between retries. Default 1 second. Use with`lookup.max-retries` parameter. [...] -| http.source.lookup.retry-strategy.exponential-delay.initial-backoff | optional | Exponential-delay initial delay. Default 1 second. [...] -| http.source.lookup.retry-strategy.exponential-delay.max-backoff | optional | Exponential-delay maximum delay. Default 1 minute. Use with `lookup.max-retries` parameter. [...] -| http.source.lookup.retry-strategy.exponential-delay.backoff-multiplier | optional | Exponential-delay multiplier. Default value 1.5 [...] -| http.source.lookup.proxy.host | optional | Specify the hostname of the proxy. [...] -| http.source.lookup.proxy.port | optional | Specify the port of the proxy. [...] -| http.source.lookup.proxy.username | optional | Specify the username used for proxy authentication. [...] -| http.source.lookup.proxy.password | optional | Specify the password used for proxy authentication. [...] -| http.request.query-param-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to query parameters. The parameters are separated by semicolons, such as `param1;param2`. [...] -| http.request.body-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to the body. The parameters are separated by semicolons, such as `param1;param2`. [...] -| http.request.additional-body-json | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. Additional JSON content to be merged into the request body for PUT and POST operations. The value should be a valid JSON object string (e.g., `'{"opportunity":{"source":"flink"},"priority":1}'`) that will be parsed and its fields merged at the top level with the generated request body. For example, if the body is `{"id":123}` and additional-bod [...] +| Option | Required | Description/Value [...] +|:-----------------------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...] +| connector | required | The Value should be set to _http_ [...] +| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. [...] +| url | required | The base URL that should be used for GET requests. For example _http://localhost:8080/client_ [...] +| asyncPolling | optional | true/false - determines whether Async Polling should be used. Mechanism is based on Flink's Async I/O. [...] +| lookup-method | optional | GET/POST/PUT (and any other) - determines what REST method should be used for lookup REST query. If not specified, `GET` method will be used. [...] +| lookup.cache | optional | Enum possible values: `NONE`, `PARTIAL`. The cache strategy for the lookup table. Currently supports `NONE` (no caching) and `PARTIAL` (caching entries on lookup operation in external API). [...] +| lookup.partial-cache.max-rows | optional | The max number of rows of lookup cache, over this value, the oldest rows will be expired. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] +| lookup.partial-cache.expire-after-write | optional | The max time to live for each rows in lookup cache after writing into the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] +| lookup.partial-cache.expire-after-access | optional | The max time to live for each rows in lookup cache after accessing the entry in the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] +| lookup.partial-cache.cache-missing-key | optional | This is a boolean that defaults to true. Whether to store an empty value into the cache if the lookup key doesn't match any rows in the table. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] +| lookup.max-retries | optional | The max retry times if the lookup failed; default is 3. See the following <a href="#lookup-cache">Lookup Cache</a> section for more detail. Set value 0 to disable retries. [...] +| http.security.cert.server | optional | Comma separated paths to trusted HTTP server certificates that should be added to the connectors trust store. [...] +| http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. [...] +| http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. [...] +| http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. [...] +| http.security.oidc.token.request | optional | OIDC `Token Request` body in `application/x-www-form-urlencoded` encoding [...] +| http.security.oidc.token.endpoint.url | optional | OIDC `Token Endpoint` url, to which the token request will be issued [...] +| http.security.oidc.token.expiry.reduction | optional | OIDC tokens will be requested if the current time is later than the cached token expiry time minus this value. [...] +| http.source.lookup.continue-on-error | optional | When true, the flow will continue on errors, returning row content. When false (the default) the job ends on errors. [...] +| http.source.lookup.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. [...] +| http.source.lookup.request.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 8 threads will be used. [...] +| http.source.lookup.response.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup response processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 4 threads will be used. [...] +| http.source.lookup.use-raw-authorization-header | optional | If set to `'true'`, uses the raw value set for the `Authorization` header, without transformation for Basic Authentication (base64, addition of "Basic " prefix). If not specified, defaults to `'false'`. [...] +| http.source.lookup.request-callback | optional | Specify which `HttpLookupPostRequestCallback` implementation to use. By default, it is set to `slf4j-lookup-logger` corresponding to `Slf4jHttpLookupPostRequestCallback`. [...] +| http.source.lookup.connection.timeout | optional | Source table connection timeout. Default - no value. [...] +| http.source.lookup.http-version | optional | Version of HTTP to use for lookup http requests. The valid values are HTTP_1_1 and HTTP_2, which specify HTTP 1.1 or 2 respectively. This option may be required as HTTP_1_1, if the endpoint is HTTP 1.1, because some http 1.1 endpoints reject HTTP Version 2 calls, with 'Invalid HTTP request received' and 'HTTP/2 upgrade not supported'. [...] +| http.source.lookup.success-codes | optional | Comma separated http codes considered as success response. Use [1-5]XX for groups and '!' character for excluding. The default is 2XX. [...] +| http.source.lookup.retry-codes | optional | Comma separated http codes considered as transient errors. Use [1-5]XX for groups and '!' character for excluding. The default is 500,503,504. [...] +| http.source.lookup.ignored-response-codes | optional | Comma separated http codes. Content for these responses will be ignored. Use [1-5]XX for groups and '!' character for excluding. Ignored responses together with `http.source.lookup.success-codes` are considered as successful. [...] +| http.source.lookup.retry-strategy.type | optional | Auto retry strategy type: fixed-delay (default) or exponential-delay. [...] +| http.source.lookup.retry-strategy.fixed-delay.delay | optional | Fixed-delay interval between retries. Default 1 second. Use with`lookup.max-retries` parameter. [...] +| http.source.lookup.retry-strategy.exponential-delay.initial-backoff | optional | Exponential-delay initial delay. Default 1 second. [...] +| http.source.lookup.retry-strategy.exponential-delay.max-backoff | optional | Exponential-delay maximum delay. Default 1 minute. Use with `lookup.max-retries` parameter. [...] +| http.source.lookup.retry-strategy.exponential-delay.backoff-multiplier | optional | Exponential-delay multiplier. Default value 1.5 [...] +| http.source.lookup.proxy.host | optional | Specify the hostname of the proxy. [...] +| http.source.lookup.proxy.port | optional | Specify the port of the proxy. [...] +| http.source.lookup.proxy.username | optional | Specify the username used for proxy authentication. [...] +| http.source.lookup.proxy.password | optional | Specify the password used for proxy authentication. [...] +| http.request.query-param-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to query parameters. The parameters are separated by semicolons, such as `param1;param2`. [...] +| http.request.body-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to the body. The parameters are separated by semicolons, such as `param1;param2`. [...] +| http.request.additional-body-json | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. Additional JSON content to be merged into the request body for PUT and POST operations. The value should be a valid JSON object string (e.g., `'{"opportunity":{"source":"flink"},"priority":1}'`) that will be parsed and its fields merged at the top level with the generated request body. For example, if the body is `{"id":123}` and additional-bod [...] | http.request.url-map | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The map of insert names to column names used as url segments. Parses a string as a map of strings. For example if there are table columns called `customerId` and `orderId`, then specifying value `customerId:cid,orderID:oid` and a url of https://myendpoint/customers/{cid}/orders/{oid} will mean that the url used for the lookup query will dynamic [...] ### Query Creators In the above example we see that HTTP GET operations and HTTP POST operations result in different mapping of the columns to the HTTP request content. In reality, you will want to have more control over how the SQL columns are mapped to the HTTP content. -The HTTP connector supplies a number of Query Creators that you can use define these mappings. +The HTTP connector supplies a number of Query Creators that you can use to define these mappings. <table class="table table-bordered"> <thead> @@ -279,7 +279,7 @@ To create a custom format user has to implement Flink's `SerializationSchema` an `resources/META-INF.services/org.apache.flink.table.factories.Factory` file. This is common Flink mechanism for providing custom implementations for various factories. ### Http headers -It is possible to set HTTP headers that will be added to HTTP request send by lookup source connector. +It is possible to set HTTP headers that will be added to HTTP request sent by lookup source connector. Headers are defined via property key `http.source.lookup.header.HEADER_NAME = header value` for example: `http.source.lookup.header.X-Content-Type-Options = nosniff`. @@ -411,7 +411,7 @@ You can configure a list of HTTP status codes that should be treated as errors f By default all 400 and 500 response codes will be interpreted as error code. This behavior can be changed by using below properties in table definition. The property name are: -- `http.sink.error.code` used to defined HTTP status code value that should be treated as error for example 404. +- `http.sink.error.code` used to define HTTP status code value that should be treated as error for example 404. Many status codes can be defined in one value, where each code should be separated with comma, for example: `401, 402, 403`. User can use this property also to define a type code mask. In that case, all codes from given HTTP response type will be treated as errors. An example of such a mask would be `3XX, 4XX, 5XX`. In this case, all 300s, 400s and 500s status codes will be treated as errors. @@ -556,7 +556,7 @@ WITH ( 'http.source.lookup.ignored-response-codes' = '404' ) ``` -All 200s codes and 404 are considered as successful (`success-codes`, `ignored-response-codes`). These responses won't cause retry or job failure. 404 response is listed in `ignored-response-codes` parameter, what means content body will be ignored. Http with 404 code will produce just empty record. +All 200s codes and 404 are considered as successful (`success-codes`, `ignored-response-codes`). These responses won't cause retry or job failure. 404 response is listed in `ignored-response-codes` parameter, which means content body will be ignored. Http with 404 code will produce just empty record. When server returns response with 500s code except 501, 505 and 506 then connector will re-send request based on configuration in `http.source.lookup.retry-strategy` parameters. By default it's fixed-delay with 1 second delay, up to 3 times per request (parameter `lookup.max-retries`). After exceeding max-retries limit the job will fail. A response with any other code than specified in params `success-codes` and `retry-codes` e.g. 400, 505, 301 will cause job failure. @@ -590,7 +590,7 @@ used - allowing use of globally recognized CAs without the need for configuratio You can also configure the connector to use mTLS. For this simply use `http.security.cert.client` and `http.security.key.client` connector properties to specify paths to the certificate and -private key. The key MUST be in `PCKS8` format. Both PEM and DER keys are +private key. The key MUST be in `PKCS8` format. Both PEM and DER keys are allowed. For non production environments it is sometimes necessary to use HTTPS connection and accept all certificates. @@ -622,7 +622,7 @@ that HTTP responses or an exception has occurred (for example connection Refused Logging HTTP may not be appropriate for production systems; where sensitive information is not allowed into the logs. But in development environments it is useful to be able to see HTTP content. Sensitive information can occur in the headers for example authentication tokens and passwords. Also the HTTP request and response bodies -could sensitive. The default minimal logging should be used in production. For development, you can specify config option `http.logging.level`. +could be sensitive. The default minimal logging should be used in production. For development, you can specify config option `http.logging.level`. This dictates the amount of content that debug logging will show around HTTP calls; the valid values are: | log level | Request method | URI | HTTP Body | Response status code | Headers | @@ -637,5 +637,5 @@ Notes: #### Restrictions at this time * No authentication is applied to the token request. -* The processing does not use the refresh token if it present. +* The processing does not use the refresh token if it is present. {{< top >}} diff --git a/docs/content/docs/connectors/datastream/http.md b/docs/content/docs/connectors/datastream/http.md index c1e7499..f56c848 100644 --- a/docs/content/docs/connectors/datastream/http.md +++ b/docs/content/docs/connectors/datastream/http.md @@ -120,7 +120,7 @@ HttpSink.<String>builder() or ```java -Properties properties = Properties(); +Properties properties = new Properties(); properties.setProperty("http.sink.header.X-Content-Type-Options", "nosniff"); HttpSink.<String>builder() @@ -146,7 +146,7 @@ used - allowing use of globally recognized CAs without the need for configuratio You can also configure the connector to use mTLS. For this simply use `flink.connector.http.security.cert.client` and `flink.connector.http.security.key.client` connector properties to specify paths to the certificate and -private key. The key MUST be in `PCKS8` format. Both PEM and DER keys are +private key. The key MUST be in `PKCS8` format. Both PEM and DER keys are allowed. All properties can be set via Sink's builder `.setProperty(...)` method or through Sink and Source table DDL. @@ -162,7 +162,7 @@ If the used value starts with the prefix `Basic`, or `flink.connector.http.sourc is set to `'true'`, it will be used as header value as is, without any extra modification. ## OIDC Bearer Authentication -The connector supports Bearer Authentication using a HTTP `Authorization` header. The [OAuth 2.0 rcf](https://datatracker.ietf.org/doc/html/rfc6749) mentions [Obtaining Authorization](https://datatracker.ietf.org/doc/html/rfc6749#section-4) +The connector supports Bearer Authentication using a HTTP `Authorization` header. The [OAuth 2.0 RFC](https://datatracker.ietf.org/doc/html/rfc6749) mentions [Obtaining Authorization](https://datatracker.ietf.org/doc/html/rfc6749#section-4) and an authorization grant. OIDC makes use of this [authorisation grant](https://datatracker.ietf.org/doc/html/rfc6749#section-1.3) in a [Token Request](https://openid.net/specs/openid-connect-core-1_0.html#TokenRequest) by including a [OAuth grant type](https://oauth.net/2/grant-types/) and associated properties, the response is the [token response](https://openid.net/specs/openid-connect-core-1_0.html#TokenResponse). If you want to use this authorization then you should supply the `Token Request` body in `application/x-www-form-urlencoded` encoding @@ -175,5 +175,5 @@ be requested if the current time is later than the cached token expiry time minu ### Restrictions at this time * No authentication is applied to the token request. -* The processing does not use the refresh token if it present. +* The processing does not use the refresh token if it is present. {{< top >}} diff --git a/docs/content/docs/connectors/table/http.md b/docs/content/docs/connectors/table/http.md index 9444b83..d919359 100644 --- a/docs/content/docs/connectors/table/http.md +++ b/docs/content/docs/connectors/table/http.md @@ -163,56 +163,56 @@ Or for REST POST method they will be converted to Json and used as request body. Note the options with the prefix _http_ are the HTTP connector specific options, the others are Flink options. -| Option | Required | Description/Value [...] -|:-----------------------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...] -| connector | required | The Value should be set to _http_ [...] -| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. [...] -| url | required | The base URL that should be use for GET requests. For example _http://localhost:8080/client_ [...] -| asyncPolling | optional | true/false - determines whether Async Polling should be used. Mechanism is based on Flink's Async I/O. [...] -| lookup-method | optional | GET/POST/PUT (and any other) - determines what REST method should be used for lookup REST query. If not specified, `GET` method will be used. [...] -| lookup.cache | optional | Enum possible values: `NONE`, `PARTIAL`. The cache strategy for the lookup table. Currently supports `NONE` (no caching) and `PARTIAL` (caching entries on lookup operation in external API). [...] -| lookup.partial-cache.max-rows | optional | The max number of rows of lookup cache, over this value, the oldest rows will be expired. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] -| lookup.partial-cache.expire-after-write | optional | The max time to live for each rows in lookup cache after writing into the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] -| lookup.partial-cache.expire-after-access | optional | The max time to live for each rows in lookup cache after accessing the entry in the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] -| lookup.partial-cache.cache-missing-key | optional | This is a boolean that defaults to true. Whether to store an empty value into the cache if the lookup key doesn't match any rows in the table. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] -| lookup.max-retries | optional | The max retry times if the lookup failed; default is 3. See the following <a href="#lookup-cache">Lookup Cache</a> section for more detail. Set value 0 to disable retries. [...] -| http.security.cert.server | optional | Comma separated paths to trusted HTTP server certificates that should be added to the connectors trust store. [...] -| http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. [...] -| http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. [...] -| http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. [...] -| http.security.oidc.token.request | optional | OIDC `Token Request` body in `application/x-www-form-urlencoded` encoding [...] -| http.security.oidc.token.endpoint.url | optional | OIDC `Token Endpoint` url, to which the token request will be issued [...] -| http.security.oidc.token.expiry.reduction | optional | OIDC tokens will be requested if the current time is later than the cached token expiry time minus this value. [...] -| http.source.lookup.continue-on-error | optional | When true, the flow will continue on errors, returning row content. When false (the default) the job ends on errors. [...] -| http.source.lookup.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. [...] -| http.source.lookup.request.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 8 threads will be used. [...] -| http.source.lookup.response.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup response processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 4 threads will be used. [...] -| http.source.lookup.use-raw-authorization-header | optional | If set to `'true'`, uses the raw value set for the `Authorization` header, without transformation for Basic Authentication (base64, addition of "Basic " prefix). If not specified, defaults to `'false'`. [...] -| http.source.lookup.request-callback | optional | Specify which `HttpLookupPostRequestCallback` implementation to use. By default, it is set to `slf4j-lookup-logger` corresponding to `Slf4jHttpLookupPostRequestCallback`. [...] -| http.source.lookup.connection.timeout | optional | Source table connection timeout. Default - no value. [...] -| http.source.lookup.http-version | optional | Version of HTTP to use for lookup http requests. The valid values are HTTP_1_1 and HTTP_2, which specify HTTP 1.1 or 2 respectively. This option may be required as HTTP_1_1, if the endpoint is HTTP 1.1, because some http 1.1 endpoints reject HTTP Version 2 calls, with 'Invalid HTTP request received' and 'HTTP/2 upgrade not supported'. [...] -| http.source.lookup.success-codes | optional | Comma separated http codes considered as success response. Use [1-5]XX for groups and '!' character for excluding. The default is 2XX. [...] -| http.source.lookup.retry-codes | optional | Comma separated http codes considered as transient errors. Use [1-5]XX for groups and '!' character for excluding. The default is 500,503,504. [...] -| http.source.lookup.ignored-response-codes | optional | Comma separated http codes. Content for these responses will be ignored. Use [1-5]XX for groups and '!' character for excluding. Ignored responses together with `http.source.lookup.success-codes` are considered as successful. [...] -| http.source.lookup.retry-strategy.type | optional | Auto retry strategy type: fixed-delay (default) or exponential-delay. [...] -| http.source.lookup.retry-strategy.fixed-delay.delay | optional | Fixed-delay interval between retries. Default 1 second. Use with`lookup.max-retries` parameter. [...] -| http.source.lookup.retry-strategy.exponential-delay.initial-backoff | optional | Exponential-delay initial delay. Default 1 second. [...] -| http.source.lookup.retry-strategy.exponential-delay.max-backoff | optional | Exponential-delay maximum delay. Default 1 minute. Use with `lookup.max-retries` parameter. [...] -| http.source.lookup.retry-strategy.exponential-delay.backoff-multiplier | optional | Exponential-delay multiplier. Default value 1.5 [...] -| http.source.lookup.proxy.host | optional | Specify the hostname of the proxy. [...] -| http.source.lookup.proxy.port | optional | Specify the port of the proxy. [...] -| http.source.lookup.proxy.username | optional | Specify the username used for proxy authentication. [...] -| http.source.lookup.proxy.password | optional | Specify the password used for proxy authentication. [...] -| http.request.query-param-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to query parameters. The parameters are separated by semicolons, such as `param1;param2`. [...] -| http.request.body-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to the body. The parameters are separated by semicolons, such as `param1;param2`. [...] -| http.request.additional-body-json | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. Additional JSON content to be merged into the request body for PUT and POST operations. The value should be a valid JSON object string (e.g., `'{"opportunity":{"source":"flink"},"priority":1}'`) that will be parsed and its fields merged at the top level with the generated request body. For example, if the body is `{"id":123}` and additional-bod [...] +| Option | Required | Description/Value [...] +|:-----------------------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...] +| connector | required | The Value should be set to _http_ [...] +| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. [...] +| url | required | The base URL that should be used for GET requests. For example _http://localhost:8080/client_ [...] +| asyncPolling | optional | true/false - determines whether Async Polling should be used. Mechanism is based on Flink's Async I/O. [...] +| lookup-method | optional | GET/POST/PUT (and any other) - determines what REST method should be used for lookup REST query. If not specified, `GET` method will be used. [...] +| lookup.cache | optional | Enum possible values: `NONE`, `PARTIAL`. The cache strategy for the lookup table. Currently supports `NONE` (no caching) and `PARTIAL` (caching entries on lookup operation in external API). [...] +| lookup.partial-cache.max-rows | optional | The max number of rows of lookup cache, over this value, the oldest rows will be expired. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] +| lookup.partial-cache.expire-after-write | optional | The max time to live for each rows in lookup cache after writing into the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] +| lookup.partial-cache.expire-after-access | optional | The max time to live for each rows in lookup cache after accessing the entry in the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] +| lookup.partial-cache.cache-missing-key | optional | This is a boolean that defaults to true. Whether to store an empty value into the cache if the lookup key doesn't match any rows in the table. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details. [...] +| lookup.max-retries | optional | The max retry times if the lookup failed; default is 3. See the following <a href="#lookup-cache">Lookup Cache</a> section for more detail. Set value 0 to disable retries. [...] +| http.security.cert.server | optional | Comma separated paths to trusted HTTP server certificates that should be added to the connectors trust store. [...] +| http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. [...] +| http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. [...] +| http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. [...] +| http.security.oidc.token.request | optional | OIDC `Token Request` body in `application/x-www-form-urlencoded` encoding [...] +| http.security.oidc.token.endpoint.url | optional | OIDC `Token Endpoint` url, to which the token request will be issued [...] +| http.security.oidc.token.expiry.reduction | optional | OIDC tokens will be requested if the current time is later than the cached token expiry time minus this value. [...] +| http.source.lookup.continue-on-error | optional | When true, the flow will continue on errors, returning row content. When false (the default) the job ends on errors. [...] +| http.source.lookup.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. [...] +| http.source.lookup.request.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 8 threads will be used. [...] +| http.source.lookup.response.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup response processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 4 threads will be used. [...] +| http.source.lookup.use-raw-authorization-header | optional | If set to `'true'`, uses the raw value set for the `Authorization` header, without transformation for Basic Authentication (base64, addition of "Basic " prefix). If not specified, defaults to `'false'`. [...] +| http.source.lookup.request-callback | optional | Specify which `HttpLookupPostRequestCallback` implementation to use. By default, it is set to `slf4j-lookup-logger` corresponding to `Slf4jHttpLookupPostRequestCallback`. [...] +| http.source.lookup.connection.timeout | optional | Source table connection timeout. Default - no value. [...] +| http.source.lookup.http-version | optional | Version of HTTP to use for lookup http requests. The valid values are HTTP_1_1 and HTTP_2, which specify HTTP 1.1 or 2 respectively. This option may be required as HTTP_1_1, if the endpoint is HTTP 1.1, because some http 1.1 endpoints reject HTTP Version 2 calls, with 'Invalid HTTP request received' and 'HTTP/2 upgrade not supported'. [...] +| http.source.lookup.success-codes | optional | Comma separated http codes considered as success response. Use [1-5]XX for groups and '!' character for excluding. The default is 2XX. [...] +| http.source.lookup.retry-codes | optional | Comma separated http codes considered as transient errors. Use [1-5]XX for groups and '!' character for excluding. The default is 500,503,504. [...] +| http.source.lookup.ignored-response-codes | optional | Comma separated http codes. Content for these responses will be ignored. Use [1-5]XX for groups and '!' character for excluding. Ignored responses together with `http.source.lookup.success-codes` are considered as successful. [...] +| http.source.lookup.retry-strategy.type | optional | Auto retry strategy type: fixed-delay (default) or exponential-delay. [...] +| http.source.lookup.retry-strategy.fixed-delay.delay | optional | Fixed-delay interval between retries. Default 1 second. Use with`lookup.max-retries` parameter. [...] +| http.source.lookup.retry-strategy.exponential-delay.initial-backoff | optional | Exponential-delay initial delay. Default 1 second. [...] +| http.source.lookup.retry-strategy.exponential-delay.max-backoff | optional | Exponential-delay maximum delay. Default 1 minute. Use with `lookup.max-retries` parameter. [...] +| http.source.lookup.retry-strategy.exponential-delay.backoff-multiplier | optional | Exponential-delay multiplier. Default value 1.5 [...] +| http.source.lookup.proxy.host | optional | Specify the hostname of the proxy. [...] +| http.source.lookup.proxy.port | optional | Specify the port of the proxy. [...] +| http.source.lookup.proxy.username | optional | Specify the username used for proxy authentication. [...] +| http.source.lookup.proxy.password | optional | Specify the password used for proxy authentication. [...] +| http.request.query-param-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to query parameters. The parameters are separated by semicolons, such as `param1;param2`. [...] +| http.request.body-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to the body. The parameters are separated by semicolons, such as `param1;param2`. [...] +| http.request.additional-body-json | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. Additional JSON content to be merged into the request body for PUT and POST operations. The value should be a valid JSON object string (e.g., `'{"opportunity":{"source":"flink"},"priority":1}'`) that will be parsed and its fields merged at the top level with the generated request body. For example, if the body is `{"id":123}` and additional-bod [...] | http.request.url-map | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The map of insert names to column names used as url segments. Parses a string as a map of strings. For example if there are table columns called `customerId` and `orderId`, then specifying value `customerId:cid,orderID:oid` and a url of https://myendpoint/customers/{cid}/orders/{oid} will mean that the url used for the lookup query will dynamic [...] ### Query Creators In the above example we see that HTTP GET operations and HTTP POST operations result in different mapping of the columns to the HTTP request content. In reality, you will want to have more control over how the SQL columns are mapped to the HTTP content. -The HTTP connector supplies a number of Query Creators that you can use define these mappings. +The HTTP connector supplies a number of Query Creators that you can use define to these mappings. <table class="table table-bordered"> <thead> @@ -279,7 +279,7 @@ To create a custom format user has to implement Flink's `SerializationSchema` an `resources/META-INF.services/org.apache.flink.table.factories.Factory` file. This is common Flink mechanism for providing custom implementations for various factories. ### Http headers -It is possible to set HTTP headers that will be added to HTTP request send by lookup source connector. +It is possible to set HTTP headers that will be added to HTTP request sent by lookup source connector. Headers are defined via property key `http.source.lookup.header.HEADER_NAME = header value` for example: `http.source.lookup.header.X-Content-Type-Options = nosniff`. @@ -411,7 +411,7 @@ You can configure a list of HTTP status codes that should be treated as errors f By default all 400 and 500 response codes will be interpreted as error code. This behavior can be changed by using below properties in table definition. The property name are: -- `http.sink.error.code` used to defined HTTP status code value that should be treated as error for example 404. +- `http.sink.error.code` used to define HTTP status code value that should be treated as error for example 404. Many status codes can be defined in one value, where each code should be separated with comma, for example: `401, 402, 403`. User can use this property also to define a type code mask. In that case, all codes from given HTTP response type will be treated as errors. An example of such a mask would be `3XX, 4XX, 5XX`. In this case, all 300s, 400s and 500s status codes will be treated as errors. @@ -556,7 +556,7 @@ WITH ( 'http.source.lookup.ignored-response-codes' = '404' ) ``` -All 200s codes and 404 are considered as successful (`success-codes`, `ignored-response-codes`). These responses won't cause retry or job failure. 404 response is listed in `ignored-response-codes` parameter, what means content body will be ignored. Http with 404 code will produce just empty record. +All 200s codes and 404 are considered as successful (`success-codes`, `ignored-response-codes`). These responses won't cause retry or job failure. 404 response is listed in `ignored-response-codes` parameter, which means content body will be ignored. Http with 404 code will produce just empty record. When server returns response with 500s code except 501, 505 and 506 then connector will re-send request based on configuration in `http.source.lookup.retry-strategy` parameters. By default it's fixed-delay with 1 second delay, up to 3 times per request (parameter `lookup.max-retries`). After exceeding max-retries limit the job will fail. A response with any other code than specified in params `success-codes` and `retry-codes` e.g. 400, 505, 301 will cause job failure. @@ -590,7 +590,7 @@ used - allowing use of globally recognized CAs without the need for configuratio You can also configure the connector to use mTLS. For this simply use `http.security.cert.client` and `http.security.key.client` connector properties to specify paths to the certificate and -private key. The key MUST be in `PCKS8` format. Both PEM and DER keys are +private key. The key MUST be in `PKCS8` format. Both PEM and DER keys are allowed. For non production environments it is sometimes necessary to use HTTPS connection and accept all certificates. @@ -622,7 +622,7 @@ that HTTP responses or an exception has occurred (for example connection Refused Logging HTTP may not be appropriate for production systems; where sensitive information is not allowed into the logs. But in development environments it is useful to be able to see HTTP content. Sensitive information can occur in the headers for example authentication tokens and passwords. Also the HTTP request and response bodies -could sensitive. The default minimal logging should be used in production. For development, you can specify config option `http.logging.level`. +could be sensitive. The default minimal logging should be used in production. For development, you can specify config option `http.logging.level`. This dictates the amount of content that debug logging will show around HTTP calls; the valid values are: | log level | Request method | URI | HTTP Body | Response status code | Headers | @@ -637,5 +637,5 @@ Notes: #### Restrictions at this time * No authentication is applied to the token request. -* The processing does not use the refresh token if it present. +* The processing does not use the refresh token if it is present. {{< top >}}
