[
https://issues.apache.org/jira/browse/FLINK-39367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-39367:
-----------------------------------
Labels: pull-request-available (was: )
> [connector-http] Add configurable error logging with detailed context
> ---------------------------------------------------------------------
>
> Key: FLINK-39367
> URL: https://issues.apache.org/jira/browse/FLINK-39367
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / HTTP
> Reporter: featzhang
> Priority: Minor
> Labels: pull-request-available
>
> h2. Background
> Currently, when the HTTP Connector encounters errors (request failures,
> unexpected status codes, etc.), the log output provides minimal information,
> making it difficult to diagnose issues in production environments.
> The existing {{http.logging.level}} mechanism controls the verbosity of
> HTTP content logging (MIN / REQ_RESP / MAX) and is always emitted at DEBUG
> level.
> While it can include request/response bodies at MAX level, it is
> fundamentally designed for development-time debugging — it logs every request
> and
> response regardless of whether an error occurred.
> This approach has two gaps for production use:
> No signal-to-noise control: In production, operators typically disable
> DEBUG logging. Errors become invisible unless someone explicitly enables
> DEBUG for
> the entire connector, which floods logs with all successful requests as
> well.
> No actionable severity: There is no way to emit error information at ERROR
> or WARN level so that log-based alerting systems (e.g. Splunk, Datadog,
> PagerDuty) can detect failures without relying on DEBUG output.
> h2. Why Not Extend {{http.logging.level}}?
> The existing mechanism maps a single level (MIN/REQ_RESP/MAX) to the amount
> of content logged. Extending it to also control the SLF4J severity at which
> errors are reported would conflate two unrelated concerns:
> - Content verbosity: how much detail to include (headers, body, etc.)
> - Error severity: at what log level to surface failures (ERROR/WARN/INFO)
> These are orthogonal dimensions. A user may want minimal content (no body)
> but still want errors logged at ERROR level for alerting. Conversely, they may
> want MAX content detail but only at DEBUG to avoid noise in a monitoring
> system.
> Introducing a separate {{http.error.log.*}} configuration set keeps these
> concerns cleanly separated and matches the pattern used in other Flink
> connectors (e.g. Kafka, JDBC) where error handling and general logging are
> configured independently.
> h2. Proposed New Configuration
> || Option || Default || Description ||
> | {{http.error.log.level}} | {{ERROR}} | SLF4J level at which HTTP errors
> are logged (ERROR / WARN / INFO) |
> | {{http.error.log.include.body}} | {{false}} | Whether to include
> request/response body in error log output |
> | {{http.error.log.body.max.size}} | {{1024}} | Maximum number of
> characters logged from body content |
> h2. Behavior with {{continue-on-error}}
> {{HttpErrorLogger}} respects the {{http.source.lookup.continue-on-error}}
> setting:
> - When {{continue-on-error=false}}: the error will be thrown, so it is
> logged at the configured {{http.error.log.level}} (ERROR/WARN/INFO) to ensure
> visibility.
> - When {{continue-on-error=true}}: the error is tolerated and processing
> continues. In this case only a DEBUG message is emitted, since logging at
> ERROR/WARN would be misleading — the connector is functioning as configured.
> h2. Relationship Between the Two Mechanisms
> || Mechanism || Purpose || Fixed Level || Scope ||
> | {{http.logging.level}} | Content verbosity for all HTTP traffic | DEBUG |
> Every request and response |
> | {{http.error.log.level}} | Error severity for failure scenarios |
> Configurable (ERROR/WARN/INFO) | Error cases only |
> Both can be configured independently. Neither replaces the other.
> h2. Security
> Sensitive headers ({{authorization}}, {{cookie}}, {{api-key}},
> {{x-api-key}}) are automatically masked in all {{HttpErrorLogger}} output
> regardless of
> configuration.
> h2. Related PR
> https://github.com/apache/flink-connector-http/pull/35
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
