[
https://issues.apache.org/jira/browse/FLINK-39367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
featzhang updated FLINK-39367:
------------------------------
Description:
h2. Background
Currently, when the HTTP Connector encounters errors (request failures,
unexpected status codes, etc.), the log output provides minimal information,
making it difficult to diagnose issues in production environments.
The existing {{http.logging.level}} mechanism controls the verbosity of HTTP
content logging (MIN / REQ_RESP / MAX) and is always emitted at DEBUG level.
While it can include request/response bodies at MAX level, it is
fundamentally designed for development-time debugging — it logs every request
and
response regardless of whether an error occurred.
This approach has two gaps for production use:
No signal-to-noise control: In production, operators typically disable DEBUG
logging. Errors become invisible unless someone explicitly enables DEBUG for
the entire connector, which floods logs with all successful requests as well.
No actionable severity: There is no way to emit error information at ERROR or
WARN level so that log-based alerting systems (e.g. Splunk, Datadog,
PagerDuty) can detect failures without relying on DEBUG output.
h2. Why Not Extend {{http.logging.level}}?
The existing mechanism maps a single level (MIN/REQ_RESP/MAX) to the amount
of content logged. Extending it to also control the SLF4J severity at which
errors are reported would conflate two unrelated concerns:
- Content verbosity: how much detail to include (headers, body, etc.)
- Error severity: at what log level to surface failures (ERROR/WARN/INFO)
These are orthogonal dimensions. A user may want minimal content (no body)
but still want errors logged at ERROR level for alerting. Conversely, they may
want MAX content detail but only at DEBUG to avoid noise in a monitoring
system.
Introducing a separate {{http.error.log.*}} configuration set keeps these
concerns cleanly separated and matches the pattern used in other Flink
connectors (e.g. Kafka, JDBC) where error handling and general logging are
configured independently.
h2. Proposed New Configuration
|| Option || Default || Description ||
| {{http.error.log.level}} | {{ERROR}} | SLF4J level at which HTTP errors are
logged (ERROR / WARN / INFO) |
| {{http.error.log.include.body}} | {{false}} | Whether to include
request/response body in error log output |
| {{http.error.log.body.max.size}} | {{1024}} | Maximum number of characters
logged from body content |
h2. Behavior with {{continue-on-error}}
{{HttpErrorLogger}} respects the {{http.source.lookup.continue-on-error}}
setting:
- When {{continue-on-error=false}}: the error will be thrown, so it is logged
at the configured {{http.error.log.level}} (ERROR/WARN/INFO) to ensure
visibility.
- When {{continue-on-error=true}}: the error is tolerated and processing
continues. In this case only a DEBUG message is emitted, since logging at
ERROR/WARN would be misleading — the connector is functioning as configured.
h2. Relationship Between the Two Mechanisms
|| Mechanism || Purpose || Fixed Level || Scope ||
| {{http.logging.level}} | Content verbosity for all HTTP traffic | DEBUG |
Every request and response |
| {{http.error.log.level}} | Error severity for failure scenarios |
Configurable (ERROR/WARN/INFO) | Error cases only |
Both can be configured independently. Neither replaces the other.
h2. Security
Sensitive headers ({{authorization}}, {{cookie}}, {{api-key}}, {{x-api-key}})
are automatically masked in all {{HttpErrorLogger}} output regardless of
configuration.
h2. Related PR
https://github.com/apache/flink-connector-http/pull/35
was:
h2. Background
Currently, when the HTTP Connector encounters errors (request failures,
unexpected status codes, etc.), the log output provides minimal information,
making it difficult to diagnose issues in production environments. The
existing {{http.logging.level}} mechanism only controls the verbosity of
request/response content logging (MIN/REQ_RESP/MAX) and is fixed at DEBUG
level, which does not address the following needs:
- Logging errors at WARN or INFO level instead of only DEBUG
- Capturing richer context at error time (HTTP method, URL, response status
code, retry attempt number)
- Optionally including request/response bodies in error logs, with a
configurable size limit to prevent log flooding
- Automatically masking sensitive headers (e.g., Authorization, Cookie,
api-key) to avoid credential exposure
h2. Requirements
Introduce a new {{HttpErrorLogger}} class providing configurable error log
level and verbosity
Add the following three new configuration options:
** {{http.error.log.level}} — log level for error events; one of
{{ERROR}}/{{WARN}}/{{INFO}}, default {{ERROR}}
** {{http.error.log.include.body}} — whether to include request/response body
in error logs, default {{false}}
** {{http.error.log.body.max.size}} — maximum number of characters to log
from body content, default {{1024}}
Sensitive headers (authorization, cookie, api-key, x-api-key) must be
automatically masked in all error log output
Refactor {{GenericJsonAndUrlQueryCreator}} to replace the regex-based
{{http.request.body-template}} with a more explicit and type-safe approach using
{{http.request.body-fields}} and {{http.request.additional-body-json}}
h2. Proposed Solution
- Add {{HttpErrorLogLevel}} enum (ERROR / WARN / INFO)
- Add {{HttpErrorLogger}} class implementing {{Serializable}} for use in
distributed Flink environments; reads configuration from {{Properties}} at
construction time
- Integrate {{HttpErrorLogger}} into both {{JavaNetHttpPollingClient}}
(lookup source) and {{JavaNetSinkHttpClient}} (sink)
- Refactor {{GenericJsonAndUrlQueryCreatorFactory}}:
** Replace {{http.request.body-template}} (regex placeholder substitution)
with:
*** {{http.request.body-fields}}: explicit list of column names to include in
the request body for PUT/POST operations
*** {{http.request.additional-body-json}}: a static JSON object string whose
fields are merged into the generated request body; parsed once at factory
initialization for runtime efficiency
** Validate at factory creation time that {{additional-body-json}} fields do
not conflict with join key fields defined in {{body-fields}}
h2. Notes
{{HttpErrorLogger}} and the existing {{HttpLogger}} serve complementary
purposes:
- {{HttpLogger}}: controls the verbosity of HTTP content logging (MIN /
REQ_RESP / MAX), always at DEBUG level — intended for development and
troubleshooting
- {{HttpErrorLogger}}: controls the severity level at which errors are logged
(ERROR / WARN / INFO) — intended for production operational monitoring
These two mechanisms are independent and do not conflict. Users may configure
both as needed.
h2. Related PR
https://github.com/apache/flink-connector-http/pull/35
> [connector-http] Add configurable error logging with detailed context
> ---------------------------------------------------------------------
>
> Key: FLINK-39367
> URL: https://issues.apache.org/jira/browse/FLINK-39367
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / HTTP
> Reporter: featzhang
> Priority: Minor
>
> h2. Background
> Currently, when the HTTP Connector encounters errors (request failures,
> unexpected status codes, etc.), the log output provides minimal information,
> making it difficult to diagnose issues in production environments.
> The existing {{http.logging.level}} mechanism controls the verbosity of
> HTTP content logging (MIN / REQ_RESP / MAX) and is always emitted at DEBUG
> level.
> While it can include request/response bodies at MAX level, it is
> fundamentally designed for development-time debugging — it logs every request
> and
> response regardless of whether an error occurred.
> This approach has two gaps for production use:
> No signal-to-noise control: In production, operators typically disable
> DEBUG logging. Errors become invisible unless someone explicitly enables
> DEBUG for
> the entire connector, which floods logs with all successful requests as
> well.
> No actionable severity: There is no way to emit error information at ERROR
> or WARN level so that log-based alerting systems (e.g. Splunk, Datadog,
> PagerDuty) can detect failures without relying on DEBUG output.
> h2. Why Not Extend {{http.logging.level}}?
> The existing mechanism maps a single level (MIN/REQ_RESP/MAX) to the amount
> of content logged. Extending it to also control the SLF4J severity at which
> errors are reported would conflate two unrelated concerns:
> - Content verbosity: how much detail to include (headers, body, etc.)
> - Error severity: at what log level to surface failures (ERROR/WARN/INFO)
> These are orthogonal dimensions. A user may want minimal content (no body)
> but still want errors logged at ERROR level for alerting. Conversely, they may
> want MAX content detail but only at DEBUG to avoid noise in a monitoring
> system.
> Introducing a separate {{http.error.log.*}} configuration set keeps these
> concerns cleanly separated and matches the pattern used in other Flink
> connectors (e.g. Kafka, JDBC) where error handling and general logging are
> configured independently.
> h2. Proposed New Configuration
> || Option || Default || Description ||
> | {{http.error.log.level}} | {{ERROR}} | SLF4J level at which HTTP errors
> are logged (ERROR / WARN / INFO) |
> | {{http.error.log.include.body}} | {{false}} | Whether to include
> request/response body in error log output |
> | {{http.error.log.body.max.size}} | {{1024}} | Maximum number of
> characters logged from body content |
> h2. Behavior with {{continue-on-error}}
> {{HttpErrorLogger}} respects the {{http.source.lookup.continue-on-error}}
> setting:
> - When {{continue-on-error=false}}: the error will be thrown, so it is
> logged at the configured {{http.error.log.level}} (ERROR/WARN/INFO) to ensure
> visibility.
> - When {{continue-on-error=true}}: the error is tolerated and processing
> continues. In this case only a DEBUG message is emitted, since logging at
> ERROR/WARN would be misleading — the connector is functioning as configured.
> h2. Relationship Between the Two Mechanisms
> || Mechanism || Purpose || Fixed Level || Scope ||
> | {{http.logging.level}} | Content verbosity for all HTTP traffic | DEBUG |
> Every request and response |
> | {{http.error.log.level}} | Error severity for failure scenarios |
> Configurable (ERROR/WARN/INFO) | Error cases only |
> Both can be configured independently. Neither replaces the other.
> h2. Security
> Sensitive headers ({{authorization}}, {{cookie}}, {{api-key}},
> {{x-api-key}}) are automatically masked in all {{HttpErrorLogger}} output
> regardless of
> configuration.
> h2. Related PR
> https://github.com/apache/flink-connector-http/pull/35
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
