[
https://issues.apache.org/jira/browse/UNOMI-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132617#comment-18132617
]
Jerome Blanchard commented on UNOMI-924:
----------------------------------------
Can this fix be backported to the 3.0.x line, with 3.0.2 added as a fix version?
The fix from apache/unomi#746 (https://github.com/apache/unomi/pull/746, commit
0c44c9bd) is only on master, and the first tag that contains it is
unomi-root-4.0.0-rc1. The unomi-3.0.x branch (3.0.2-SNAPSHOT) still has the old
HealthCheckService. In 3.0.1, a second call that arrives while a check runs
gets a RejectedExecutionException, and the servlet returns HTTP 500. A
Kubernetes probe or a load balancer that calls /health/check in parallel gets
these errors on a healthy node.
When we read the change, we found a point that we suggest to fix in the
backport, and on master:
* The change removes the timeout. HealthCheckService calls provider.execute()
directly, and the "timeout" configuration key is no longer read. The HTTP
client of the ElasticSearch and OpenSearch providers has no connect timeout and
no socket timeout (HttpUtils.initHttpClient uses an empty RequestConfig).
* If the search engine stops to answer but keeps the TCP connection open, the
refresh thread waits with no limit and "busy" stays true. Every other call then
gets the last cached result with no age limit. If that result was LIVE, the
endpoint returns HTTP 200 while the search engine does not answer.
* Before the first refresh completes, every call waits on cacheLock.
Two small changes would fix this:
# Put a timeout back on each provider call, with the configured "timeout"
value. An executor with Future.get(timeout) does this, or connect and socket
timeouts on the RequestConfig of the providers.
# Treat the cache as expired after a maximum age, also while a refresh runs,
and return the timed-out status for the providers in that case.
Two minor points, for information:
* Two threads can pass shouldRefreshCache() at the same time. The second thread
then runs a second refresh after the first, because it does not check the cache
age again inside cacheLock.
* The providers list is an ArrayList that bind() and unbind() change while
refreshCache() reads it.
We can provide the backport PR on unomi-3.0.x if that helps.
> Impossible to call healthcheck concurrently
> -------------------------------------------
>
> Key: UNOMI-924
> URL: https://issues.apache.org/jira/browse/UNOMI-924
> Project: Apache Unomi
> Issue Type: Bug
> Reporter: Jerome Blanchard
> Priority: Major
> Fix For: unomi-4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> To avoid heavy load for healthcheck, a lock has been placed to raise an
> exception when a healthcheck is already running.
> This strategy seems good at the beginning but in production we are facing
> concurrent calls to healthcheck from multiple HA Proxy nodes. As the lock
> generated an HTTP error 500, HAProxy mark the unomi node as failing.
> The goal is to fix that behavior by allowing concurrent calls but having only
> one thread calculating healthcheck and a caching strategy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
