[
https://issues.apache.org/jira/browse/UIMA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry Cwiklik updated UIMA-1902:
--------------------------------
Description:
UIMA AS service checks for existence of a reply queue before processing an
incoming request. This is done to prevent wasting cycles processing a request
if it is known that the client has gone away. Current approach is based on
checking broker's queue inventory for presence of a temp reply queue provided
in a request msg. The framework code uses JMX connection to a broker to
perform the queue lookup. If the lookup fails, the framework concludes that the
client has gone away, places the reply queue in a list of destinations known to
be dead and throws away the request msg. Any subsequent msg containing the same
reply queue is automatically thrown away since the destination is marked as
dead. Occasionally, the JMX lookup fails even though (as reported by some
users) the client is alive at the time the lookup was done. It seems that
perhaps under load JMX fails, reporting false positive result ( queue not
present). The failed lookup results in a dropped request msg and ultimately in
a client timeout (or possible hang if the client is not using timout).
Modify the implementation to stop using JMX to perform lookups. Instead, before
any processing takes place, send a small msg (ACK) back to the client via JMS.
If the send fails, report the problem, drop the request and wait for the next
request. Also, dont add failed destination to the list of clients known to be
dead. Each request should be subject to an ACK no matter what happened
previously.
The new approach, although generating more jms traffic, has some advantages:
1) A client is immediately notified when its request has been picked by a
service and is ready to be processed.
2) A client "knows" which service is processing each request
3) A client can use a different timeout strategy since it knows precisely when
its request is ready to be processed.
was:
UIMA AS service checks for existence of a reply queue before processing an
incoming request. This is done to prevent wasting cycles processing a request
if it is known that the client has gone away. Current approach is based on
checking broker's queue inventory for presence of a temp reply queue provided
in a request msg. The framework code uses JMX connection to a broker to
perform the queue lookup. If the lookup fails, the framework concludes that the
client has gone away, places the reply queue in a list of destinations known to
be dead and throws away the request msg. Any subsequent msg containing the same
reply queue is automatically thrown away since the destination is marked as
dead. Occasionally, the JMX lookup fails even though (as reported by some
users) the client is alive at the time the lookup was done. It seems that
perhaps under load JMX fails apart, reporting false positive result ( queue not
present). The failed lookup results in a dropped request msg and ultimately in
a client timeout (or possible hang if the client is not using timout).
Modify the implementation to stop using JMX to perform lookups. Instead, before
any processing takes place, send a small msg (ACK) back to the client via JMS.
If the send fails, report the problem, drop the request and wait for the next
request. Also, dont add failed destination to the list of clients known to be
dead. Each request should be subject to an ACK no matter what happened
previously.
The new approach, although generating more jms traffic, has some advantages:
1) A client is immediately notified when its request has been picked by a
service and is ready to be processed.
2) A client "knows" which service is processing each request
3) A client can use a different timeout strategy since it knows precisely when
its request is ready to be processed.
> UIMA AS detection of missing client reply queue is not working leading to
> client timeouts
> -----------------------------------------------------------------------------------------
>
> Key: UIMA-1902
> URL: https://issues.apache.org/jira/browse/UIMA-1902
> Project: UIMA
> Issue Type: Bug
> Components: Async Scaleout
> Affects Versions: 2.3AS
> Reporter: Jerry Cwiklik
> Assignee: Jerry Cwiklik
>
> UIMA AS service checks for existence of a reply queue before processing an
> incoming request. This is done to prevent wasting cycles processing a request
> if it is known that the client has gone away. Current approach is based on
> checking broker's queue inventory for presence of a temp reply queue provided
> in a request msg. The framework code uses JMX connection to a broker to
> perform the queue lookup. If the lookup fails, the framework concludes that
> the client has gone away, places the reply queue in a list of destinations
> known to be dead and throws away the request msg. Any subsequent msg
> containing the same reply queue is automatically thrown away since the
> destination is marked as dead. Occasionally, the JMX lookup fails even though
> (as reported by some users) the client is alive at the time the lookup was
> done. It seems that perhaps under load JMX fails, reporting false positive
> result ( queue not present). The failed lookup results in a dropped request
> msg and ultimately in a client timeout (or possible hang if the client is not
> using timout).
> Modify the implementation to stop using JMX to perform lookups. Instead,
> before any processing takes place, send a small msg (ACK) back to the client
> via JMS. If the send fails, report the problem, drop the request and wait for
> the next request. Also, dont add failed destination to the list of clients
> known to be dead. Each request should be subject to an ACK no matter what
> happened previously.
> The new approach, although generating more jms traffic, has some advantages:
> 1) A client is immediately notified when its request has been picked by a
> service and is ready to be processed.
> 2) A client "knows" which service is processing each request
> 3) A client can use a different timeout strategy since it knows precisely
> when its request is ready to be processed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
