Federico Mariani created CAMEL-26998:
----------------------------------------
Summary: camel-core - Polling consumers: the consumer health check
stays UP while a poll hangs
Key: CAMEL-26998
URL: https://issues.apache.org/jira/browse/CAMEL-26998
Project: Camel
Issue Type: Improvement
Components: camel-core
Reporter: Federico Mariani
The health check of a polling consumer ({{ScheduledPollConsumerHealthCheck}},
which every consumer that extends {{ScheduledPollConsumer}} gets: file,
ftp/sftp, sql, jpa, mail, aws2-s3, ...) reports from what a poll leaves behind
once it is done: the error counter and the last error, which
{{ScheduledPollConsumer.doRun}} updates after {{poll()}} returned or threw, and
whether the first poll is done ({{isConsumerReady()}}). A poll that does not
return is not seen: readiness stays UP for as long as it blocks.
A poll blocks for long when a client has no read timeout, or a long one, on a
connection that went silent (a half-open TCP connection, a NAT or firewall that
drops the flow). Found with fault-injection tests (Toxiproxy blackhole and
silent drop in front of a real PostgreSQL and SFTP server): the consumer thread
waits in a socket read, with no error, no WARN, and readiness UP for the whole
fault:
* camel-sql with pgjdbc and its default {{socketTimeout=0}}: the poll waits
until TCP gives up (~15 min), or for ever;
* sftp with the default {{soTimeout}} (300 s), which is the JSch session
timeout: JSch sends a keepalive at the first timeout and drops the connection
at the second, so ~600 s.
A client timeout ({{socketTimeout}}, {{soTimeout}}) is the fix for the hang
itself, but the health check cannot tell a hung poll from a quiet one either
way. It is also UP when the very first poll hangs with
{{camel.health.initialState=UP}}, and when a consumer marks itself ready before
its poll is done ({{forceConsumerAsReady()}}: sql after the query, ftp after
the listing, CAMEL-20189).
h3. Steps to reproduce
A {{ScheduledPollConsumer}} whose {{poll()}} waits on a latch: a first poll
that succeeds, then a poll that blocks. Call the consumer health check as
readiness while the poll blocks.
h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)
UP for as long as the poll blocks.
h3. Expected
A way to have the consumer health check DOWN, with a message that says so, once
a poll has been running longer than the user expects a poll to take; UP again
when the poll is done.
h3. Proposal
* {{ScheduledPollConsumer}} keeps when the poll in progress started (one
volatile time, set when begin/poll/commit starts, as {{isPolling()}} counts
them).
* A new consumer option {{healthCheckMaxPollDuration}} on
{{ScheduledPollEndpoint}}, so every polling component has it. 0 (the default)
turns it off. When set, the consumer health check is DOWN while the poll in
progress has been running longer than that, with a message such as "Consumer
has been polling for 5m10s, longer than healthCheckMaxPollDuration (5m)". The
poll is not interrupted.
Opt-in, because a poll includes routing what it polled (a batch of files, a big
file, a slow query): a default would turn readiness DOWN for polls that are
only slow (CAMEL-20189 is about readiness and a long first poll). This is not a
threshold on results that a monitoring system could apply itself (CAMEL-17587
removed those): while a poll hangs, the health check has no failure to report
at all.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)