[CONF] Apache Qpid > Qpid Java Broker Transaction Timeouts

[confluence]

Qpid Java Broker Transaction Timeouts Page edited by Andrew Kennedy Comment: made more generic Changes (12) ... Qpid 0.11 includes a new feature to allow deployers to configure transaction timeout thresholds, and the associated actions to perform on crossing of the specified threshold. The transaction timeout mechanism is used to control broker resources when clients producing messages using transactional sessions hang or otherwise become unresponsive. The transaction timeout mechanism is used to control broker resources when clients producing messages using transactional sessions hang or otherwise become unresponsive, or simply begin a transaction and keep using it without ever calling {{Session#commit()}}. Users can choose to configure a Warn threshold, after which the identified transaction should be logged as a 'WARN' level alert as well as (importantly) a Close threshold after which the transaction and the connection it applies to will be closed. Users can choose to configure an _idleWarn_ or _openWarn_ threshold, after which the identified transaction should be logged as a *WARN* level alert as well as (more importantly) an _idleClose_ or _openClose_ threshold after which the transaction and the connection it applies to will be closed. This feature is particularly useful in environments where the owner of the broker does not have full control over the implementation of clients, such as in a shared services deployment. ... h3. Purpose This feature has been introduced to address the scenario where an open transaction can include an open transaction on the BDB store. This can have undesirable consequences, preventing the BDB store cleaning thread from rolling forward data efficiently and can result in a rapidly growing store bound only by available disk space. This feature has been introduced to address the scenario where an open transaction on the broker holds an open transaction on the persistent store. This can have undesirable consequences if the store does not time out or close long-running transactions, such as with [Oracle BDB|http://www.oracle.com/technetwork/database/berkeleydb/overview/index-093405.html]. This can can result in a rapid increase in disk usage size, bounded only by available space, due to growth of the transaction log. h3. Scope Note that only {{MessageProducer}} clients will be affected by a transaction timeout, since store transaction lifespan on a consumer only spans the execution of the commit call to {{Session#commit()}} and there is no scope for a long-lived transaction to arise. It is also important to note that the transaction timeout mechanism is purely a _JMS_ transaction timeout, and unrelated to any other timeouts in the Qpid client library and will have no impact on any RDBMS your application may utilise. h3. Effect ... h4. Broker Logging and Connection Close When the 'openWarn' _openWarn_ or 'idleWarn' _idleWarn_ specified threshold is exceeded, the broker will log a 'WARN' *WARN* level alert with details of the connection and channel on which the threshold has been exceeded, along with the age of the transaction. When the 'openClose' _openClose_ or 'idelClose' _idleClose_ specified threshold value is exceeded, the broker will throw an exception back to the client on its exception listener, connection via the {{ExceptionListener}}, log the action and then close the connection. The example broker log output shown below is where the _idleWarn_ threshold specified is lower than the _idleClose_ threshold and the broker therefore logs the idle transaction 3 times before the close threshold is triggered and the connection closed out. {noformat} ... {noformat} The second example broker log output shown below illustrates the same mechanism operating on an Oopen transaction. {noformat} ... After a Close threshold has been exceeded, the trigger client will receive this exception on its exception listener, prior to being disconnected: {noformat} org.apache.qpid.AMQConnectionClosedException: Error: Idle transaction timed out [error code 506: resource error] [error code 506: resource error] ... Full Content Transaction Timeouts User Guide Introduction Qpid 0.11 includes a new feature to allow deployers to configure transaction timeout thresholds, and the associated actions to perform on crossing of the specified threshold. The transaction timeout mechanism is used to control broker resources when clients producing messages using transactional sessions hang or otherwise become unresponsive, or simply begin a transaction and keep using it without ever calling Session#commit(). Users can choose to configure an idleWarn or openWarn threshold, after which the identified transaction should be logged as a WARN level alert as well as (more importantly) an idleClose or openClose threshold after which the transaction and the connection it applies to will be closed. This feature is particularly useful in environments where the owner of the broker does not have full control over the implementation of clients, such as in a shared services deployment. The following section provide more details on this feature and its use. Purpose This feature has been introduced to address the scenario where an open transaction on the broker holds an open transaction on the persistent store. This can have undesirable consequences if the store does not time out or close long-running transactions, such as with Oracle BDB. This can can result in a rapid increase in disk usage size, bounded only by available space, due to growth of the transaction log. Scope Note that only MessageProducer clients will be affected by a transaction timeout, since store transaction lifespan on a consumer only spans the execution of the call to Session#commit() and there is no scope for a long-lived transaction to arise. It is also important to note that the transaction timeout mechanism is purely a JMS transaction timeout, and unrelated to any other timeouts in the Qpid client library and will have no impact on any RDBMS your application may utilise. Effect Full details of configuration options are provided in the sections that follow. This section gives a brief overview of what the Transaction Timeout feature can do. Broker Logging and Connection Close When the openWarn or idleWarn specified threshold is exceeded, the broker will log a WARN level alert with details of the connection and channel on which the threshold has been exceeded, along with the age of the transaction. When the openClose or idleClose specified threshold value is exceeded, the broker will throw an exception back to the client connection via the ExceptionListener, log the action and then close the connection. The example broker log output shown below is where the idleWarn threshold specified is lower than the idleClose threshold and the broker therefore logs the idle transaction 3 times before the close threshold is triggered and the connection closed out. INFO [qpid.message] MESSAGE [Queue-housekeeping-test][con:1(guest@anonymous(29494388)/test)/ch:2] CHN-1008 : Idle Transaction : 13,116 ms WARN [apache.qpid.server.AMQChannel] IDLE TRANSACTION ALERT [con:1(guest@anonymous(29494388)/test)/ch:2] 13116 ms INFO [qpid.message] MESSAGE [Queue-housekeeping-test][con:1(guest@anonymous(29494388)/test)/ch:2] CHN-1008 : Idle Transaction : 14,116 ms WARN [apache.qpid.server.AMQChannel] IDLE TRANSACTION ALERT [con:1(guest@anonymous(29494388)/test)/ch:2] 14116 ms INFO [qpid.message] MESSAGE [Queue-housekeeping-test][con:1(guest@anonymous(29494388)/test)/ch:2] CHN-1008 : Idle Transaction : 15,118 ms WARN [apache.qpid.server.AMQChannel] IDLE TRANSACTION ALERT [con:1(guest@anonymous(29494388)/test)/ch:2] 15118 ms INFO [qpid.server.protocol.AMQProtocolSession] Closing connection due to: org.apache.qpid.AMQConnectionException: Idle transaction timed out [error code 506: resource error] INFO [qpid.message] MESSAGE [Queue-housekeeping-test][con:1(guest@anonymous(29494388)/test)/ch:2] CHN-1003 : Close The second example broker log output shown below illustrates the same mechanism operating on an open transaction. INFO [qpid.message] MESSAGE [Queue-housekeeping-test][con:2(guest@anonymous(27925944)/test)/ch:2] CHN-1007 : Open Transaction : 12,406 ms WARN [apache.qpid.server.AMQChannel] OPEN TRANSACTION ALERT [con:2(guest@anonymous(27925944)/test)/ch:2] 12406 ms INFO [qpid.message] MESSAGE [Queue-housekeeping-test][con:2(guest@anonymous(27925944)/test)/ch:2] CHN-1007 : Open Transaction : 13,406 ms WARN [apache.qpid.server.AMQChannel] OPEN TRANSACTION ALERT [con:2(guest@anonymous(27925944)/test)/ch:2] 13406 ms INFO [qpid.message] MESSAGE [Queue-housekeeping-test][con:2(guest@anonymous(27925944)/test)/ch:2] CHN-1007 : Open Transaction : 14,406 ms WARN [apache.qpid.server.AMQChannel] OPEN TRANSACTION ALERT [con:2(guest@anonymous(27925944)/test)/ch:2] 14406 ms INFO [qpid.server.protocol.AMQProtocolSession] Closing connection due to: org.apache.qpid.AMQConnectionException: Open transaction timed out [error code 506: resource error] INFO [qpid.message] MESSAGE [Queue-housekeeping-test][con:2(guest@anonymous(27925944)/test)/ch:2] CHN-1003 : Close Client Side Effect After a Close threshold has been exceeded, the trigger client will receive this exception on its exception listener, prior to being disconnected: org.apache.qpid.AMQConnectionClosedException: Error: Idle transaction timed out [error code 506: resource error] [error code 506: resource error] Any later attempt to use the connection will result in this exception being thrown: Producer: Caught an Exception: javax.jms.IllegalStateException: Object org.apache.qpid.client.AMQSession_0_8@129b0e1 has been closed javax.jms.IllegalStateException: Object org.apache.qpid.client.AMQSession_0_8@129b0e1 has been closed at org.apache.qpid.client.Closeable.checkNotClosed(Closeable.java:70) at org.apache.qpid.client.AMQSession.checkNotClosed(AMQSession.java:555) at org.apache.qpid.client.AMQSession.createBytesMessage(AMQSession.java:573) at org.apache.qpid.test.unit.transacted.TransactedProducer.runTest(TransactedProducer.java:138) at org.apache.qpid.test.unit.transacted.TransactedProducer.main(TransactedProducer.java:86) Thus clients must be able to handle this case successfully, reconnecting where required and registering an exception listener on all connections. This is critical, and must be communicated to client applications by any broker owner switching on transaction timeouts. Configuration Overview Transaction timeouts are configurable separately on each defined virtual host, using the virtualhosts.xml file. We would recommend that only warnings are configured at first, which should allow broker administrators to obtain an idea of the distribution of transaction lengths on their systems, and configure production settings appropriately for both warning and closure. Ideally establishing thresholds should be achieved in a representative UAT environment, with clients and broker running, prior to any production deployment. It is impossible to give suggested values, due to the large variation in usage depending on the applications using a broker. However, clearly transactions should not span the expected lifetime of any client application as this would indicate a hung client. When configuring warning and closure timeouts, it should be noted that these only apply to message producers that are connected to the broker, but that a timeout will cause the connection to be closed - this disconnecting all producers and consumers created on that connection. This should not be an issue for environments using Mule or Spring, where connection factories can be configured appropriately to manage a single MessageProducer object per JMS Session and Connection. Clients that use the JMS API directly should be aware that sessions managing both consumers and producers, or multiple producers, will be affected by a single producer hanging or leaving a transaction idle or open, and closed, and must take appropriate action to handle that scenario. Virtualhosts.xml Entries The JMS transaction timeouts are configured on each virtual host defined in the XML configuration files. The default values for each of the parameters is 0, indicating that that particular check is disabled. Any or all of the parameters can be set, using the desired value in milliseconds, and will be checked each time the housekeeping process runs, usually set to run every 30 seconds in standard configuration. The meaning of each property is as follows: openWarn - the time a transaction can be open for (with activity occurring on it) after which a warning alert will be issued. openClose - the time a transaction can be open for before the connection it is on is closed. idleWarn - the time a transaction can be idle for (with no activity occurring on it) after which a warning alert will be issued. idleClose - the time a transaction can be idle for before the connection it is on is closed. The virtualhosts configuration is shown below, and must occur inside the //virtualhosts/virtualhost/name/ elements: virtualhosts.xml <transactionTimeout> <openWarn>10000</openWarn> <openClose>20000</openClose> <idleWarn>5000</idleWarn> <idleClose>15000</idleClose> </transactionTimeout> Change Notification Preferences View Online | View Changes | Add Comment --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:commits-subscr...@qpid.apache.org