colinwerner opened a new issue, #2667:
URL: https://github.com/apache/activemq/issues/2667
`queryTimeout` on `<lease-database-locker>` (from `AbstractJDBCLocker`,
default `-1`) looks like the setting that stops a lease statement from hanging.
On PostgreSQL with synchronous
### What happens
1. The lease `UPDATE` (`getLeaseObtainStatement()`) commits locally on the
primary. The backend then waits for a synchronous standby to confirm (wait
event `SyncRep`).
2. When `queryTimeout` expires, the JDBC driver sends a cancel request.
PostgreSQL does not roll back a transaction that is waiting for synchronous
replication: it has already committed locally. The server stops waiting and
reports success with a `WARNING` ("canceling wait for synchronous replication
due to user request ... The transaction has already committed locally, but
might not have been replicated to the standby").
3. The driver returns `executeUpdate() == 1` and attaches the warning to the
statement. `LeaseDatabaseLocker.doStart()` treats the update as successful and
calls `keepAlive()`, which succeeds in the same way, so the broker logs
"becoming master".
4. The broker is now master on a lease that exists only on the primary.
Every later store write waits in the same synchronous-replication wait. If the
primary fails over to the standby, the replicated database has no record of the
lease.
### Reproduce
- One `postgres` container with `synchronous_commit=on` and
`synchronous_standby_names='ANY 1 (missing)'` (a standby that does not exist),
so no commit is ever confirmed.
- A broker using the JDBC store with `<lease-database-locker
queryTimeout="3"/>`: it becomes master within a few seconds. `pg_stat_activity`
shows `wait_event = 'SyncRep'`, and the PostgreSQL log shows the warning above.
- The same broker with a driver-level socket timeout (pgjdbc
`socketTimeout=5`) and no `queryTimeout`: the lease statement fails with an
exception, the locker logs `lease acquire failure` and retries, and the broker
does not become master until synchronous replication is available again.
### Why this is probably a documentation issue
`JDBCAdapter` and `PostgresqlJDBCAdapter` have no lock or lease hooks;
per-database behaviour lives in the `Statements` SQL. The lease locker can also
be used without a JDBC persistence adapter (for example with KahaDB). Checking
`Statement.getWarnings()` after the lease update and failing on any warning
would be generic and too broad, since some drivers report harmless warnings,
and matching PostgreSQL's warning text would be fragile.
### Proposal
Document on the pluggable storage lockers page that:
- `queryTimeout` cancels the statement, and on databases where a cancel can
arrive after a local commit (PostgreSQL with synchronous replication) it does
not mean the lease update failed;
- driver-level timeouts (socket/read timeout, connect/login timeout) bound
the lease statements safely, because a timeout surfaces as an exception;
- the lease (`lockAcquireSleepInterval`) must be longer than the slowest
renewal the master can make: roughly `lockKeepAlivePeriod` plus the connect
timeout plus the statement (socket) timeout.
While there, the page has the lease timings the wrong way round: it says the
lock "is retained for a period whose duration is given by
`lockKeepAlivePeriod`" and is extended "by `lockAcquireSleepInterval`". The
lease lasts `lockAcquireSleepInterval` and is renewed every
`lockKeepAlivePeriod`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
