Jerome Blanchard created UNOMI-1025:
---------------------------------------
Summary: Healthcheck: restore the provider timeout
Key: UNOMI-1025
URL: https://issues.apache.org/jira/browse/UNOMI-1025
Project: Apache Unomi
Issue Type: Bug
Components: unomi(-core)
Reporter: Jerome Blanchard
Fix For: unomi-4.0.0, unomi-3.0.2
Links: relates to UNOMI-924
UNOMI-924 (https://github.com/apache/unomi/pull/746) makes the healthcheck
endpoint accept concurrent calls. The change also removes the per-provider
timeout. HealthCheckService calls provider.execute() directly, and the
"timeout" configuration key is no longer read.
The HTTP client of the ElasticSearch and OpenSearch providers has no connect
timeout and no socket timeout, because HttpUtils.initHttpClient uses an empty
RequestConfig. If the search engine stops to answer but keeps the TCP
connection open, these problems occur:
* The refresh thread waits with no limit, and "busy" stays true.
* Every other call gets the last cached result with no age limit. If that
result was LIVE, /health/check returns HTTP 200 while the search engine does
not answer.
* Before the first refresh completes, every call waits on cacheLock with no
limit.
Two minor problems are in the same class:
* Two threads can pass shouldRefreshCache() at the same time. The second thread
then runs a second refresh after the first, because it does not check the cache
age again inside cacheLock.
* The providers list is an ArrayList that bind() and unbind() change while
refreshCache() reads it.
Proposed fix:
# Run each provider call in an executor, and wait for the result with the
configured "timeout" value. On timeout, cancel the call and return
provider.timeout().
# Set the connect and socket timeouts of the ElasticSearch and OpenSearch
providers to the same "timeout" value, so that a blocked call ends.
# Check the cache age again inside cacheLock.
# Use a CopyOnWriteArrayList for the providers list.
A unit test with a provider that never answers checks that check() returns
within the timeout and marks that provider as timed out.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
