[
https://issues.apache.org/jira/browse/CAMEL-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Federico Mariani reassigned CAMEL-26998:
----------------------------------------
Assignee: Federico Mariani
> camel-core - Polling consumers: the consumer health check stays UP while a
> poll hangs
> -------------------------------------------------------------------------------------
>
> Key: CAMEL-26998
> URL: https://issues.apache.org/jira/browse/CAMEL-26998
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
>
> The health check of a polling consumer ({{ScheduledPollConsumerHealthCheck}},
> which every consumer that extends {{ScheduledPollConsumer}} gets: file,
> ftp/sftp, sql, jpa, mail, aws2-s3, ...) reports from what a poll leaves
> behind once it is done: the error counter and the last error, which
> {{ScheduledPollConsumer.doRun}} updates after {{poll()}} returned or threw,
> and whether the first poll is done ({{isConsumerReady()}}). A poll that does
> not return is not seen: readiness stays UP for as long as it blocks.
> A poll blocks for long when a client has no read timeout, or a long one, on a
> connection that went silent (a half-open TCP connection, a NAT or firewall
> that drops the flow). Found with fault-injection tests (Toxiproxy blackhole
> and silent drop in front of a real PostgreSQL and SFTP server): the consumer
> thread waits in a socket read, with no error, no WARN, and readiness UP for
> the whole fault:
> * camel-sql with pgjdbc and its default {{socketTimeout=0}}: the poll waits
> until TCP gives up (~15 min), or for ever;
> * sftp with the default {{soTimeout}} (300 s), which is the JSch session
> timeout: JSch sends a keepalive at the first timeout and drops the connection
> at the second, so ~600 s.
> A client timeout ({{socketTimeout}}, {{soTimeout}}) is the fix for the hang
> itself, but the health check cannot tell a hung poll from a quiet one either
> way. It is also UP when the very first poll hangs with
> {{camel.health.initialState=UP}}, and when a consumer marks itself ready
> before its poll is done ({{forceConsumerAsReady()}}: sql after the query, ftp
> after the listing, CAMEL-20189).
> h3. Steps to reproduce
> A {{ScheduledPollConsumer}} whose {{poll()}} waits on a latch: a first poll
> that succeeds, then a poll that blocks. Call the consumer health check as
> readiness while the poll blocks.
> h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)
> UP for as long as the poll blocks.
> h3. Expected
> A way to have the consumer health check DOWN, with a message that says so,
> once a poll has been running longer than the user expects a poll to take; UP
> again when the poll is done.
> h3. Proposal
> * {{ScheduledPollConsumer}} keeps when the poll in progress started (one
> volatile time, set when begin/poll/commit starts, as {{isPolling()}} counts
> them).
> * A new consumer option {{healthCheckMaxPollDuration}} on
> {{ScheduledPollEndpoint}}, so every polling component has it. 0 (the default)
> turns it off. When set, the consumer health check is DOWN while the poll in
> progress has been running longer than that, with a message such as "Consumer
> has been polling for 5m10s, longer than healthCheckMaxPollDuration (5m)". The
> poll is not interrupted.
> Opt-in, because a poll includes routing what it polled (a batch of files, a
> big file, a slow query): a default would turn readiness DOWN for polls that
> are only slow (CAMEL-20189 is about readiness and a long first poll). This is
> not a threshold on results that a monitoring system could apply itself
> (CAMEL-17587 removed those): while a poll hangs, the health check has no
> failure to report at all.
> _Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)