Federico Mariani created CAMEL-26998:
----------------------------------------

             Summary: camel-core - Polling consumers: the consumer health check 
stays UP while a poll hangs
                 Key: CAMEL-26998
                 URL: https://issues.apache.org/jira/browse/CAMEL-26998
             Project: Camel
          Issue Type: Improvement
          Components: camel-core
            Reporter: Federico Mariani


The health check of a polling consumer ({{ScheduledPollConsumerHealthCheck}}, 
which every consumer that extends {{ScheduledPollConsumer}} gets: file, 
ftp/sftp, sql, jpa, mail, aws2-s3, ...) reports from what a poll leaves behind 
once it is done: the error counter and the last error, which 
{{ScheduledPollConsumer.doRun}} updates after {{poll()}} returned or threw, and 
whether the first poll is done ({{isConsumerReady()}}). A poll that does not 
return is not seen: readiness stays UP for as long as it blocks.

A poll blocks for long when a client has no read timeout, or a long one, on a 
connection that went silent (a half-open TCP connection, a NAT or firewall that 
drops the flow). Found with fault-injection tests (Toxiproxy blackhole and 
silent drop in front of a real PostgreSQL and SFTP server): the consumer thread 
waits in a socket read, with no error, no WARN, and readiness UP for the whole 
fault:
* camel-sql with pgjdbc and its default {{socketTimeout=0}}: the poll waits 
until TCP gives up (~15 min), or for ever;
* sftp with the default {{soTimeout}} (300 s), which is the JSch session 
timeout: JSch sends a keepalive at the first timeout and drops the connection 
at the second, so ~600 s.

A client timeout ({{socketTimeout}}, {{soTimeout}}) is the fix for the hang 
itself, but the health check cannot tell a hung poll from a quiet one either 
way. It is also UP when the very first poll hangs with 
{{camel.health.initialState=UP}}, and when a consumer marks itself ready before 
its poll is done ({{forceConsumerAsReady()}}: sql after the query, ftp after 
the listing, CAMEL-20189).

h3. Steps to reproduce

A {{ScheduledPollConsumer}} whose {{poll()}} waits on a latch: a first poll 
that succeeds, then a poll that blocks. Call the consumer health check as 
readiness while the poll blocks.

h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)

UP for as long as the poll blocks.

h3. Expected

A way to have the consumer health check DOWN, with a message that says so, once 
a poll has been running longer than the user expects a poll to take; UP again 
when the poll is done.

h3. Proposal

* {{ScheduledPollConsumer}} keeps when the poll in progress started (one 
volatile time, set when begin/poll/commit starts, as {{isPolling()}} counts 
them).
* A new consumer option {{healthCheckMaxPollDuration}} on 
{{ScheduledPollEndpoint}}, so every polling component has it. 0 (the default) 
turns it off. When set, the consumer health check is DOWN while the poll in 
progress has been running longer than that, with a message such as "Consumer 
has been polling for 5m10s, longer than healthCheckMaxPollDuration (5m)". The 
poll is not interrupted.

Opt-in, because a poll includes routing what it polled (a batch of files, a big 
file, a slow query): a default would turn readiness DOWN for polls that are 
only slow (CAMEL-20189 is about readiness and a long first poll). This is not a 
threshold on results that a monitoring system could apply itself (CAMEL-17587 
removed those): while a poll hangs, the health check has no failure to report 
at all.

_Claude Code on behalf of Croway_




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to