Lokesh Khurana created PHOENIX-8068:
---------------------------------------
Summary: HA client: autocommit statement DML does not refresh a
stale cluster role record
Key: PHOENIX-8068
URL: https://issues.apache.org/jira/browse/PHOENIX-8068
Project: Phoenix
Issue Type: Sub-task
Reporter: Lokesh Khurana
Assignee: Lokesh Khurana
h3. Problem
With autocommit on, statement-level DML on an HA connection
({{FailoverPhoenixConnection}}) does not fail over after a consistent failover.
The server rejects each write with {{StaleClusterRoleRecordException}}. The
client does not refresh its cluster role record (CRR), so it stays on the old
active cluster. Every later write fails the same way, in a few milliseconds,
until the application opens a new connection.
The application does not get {{FAILOVER_IN_PROGRESS}} (error code 1990), so it
has no signal to call {{FailoverPhoenixConnection.failover()}} or to reconnect.
An explicit {{conn.commit()}} does not have this problem. That call goes
through {{FailoverPhoenixConnection}}, which maps the error and refreshes the
CRR.
h3. Code path
# {{FailoverPhoenixConnection.prepareStatement()}} and {{createStatement()}}
wrap only the create step. They return a raw {{PhoenixStatement}} on the
wrapped {{PhoenixConnection}}.
# {{PhoenixPreparedStatement.executeUpdate()}} calls
{{PhoenixStatement.executeMutation()}}. With autocommit on, it calls
{{PhoenixConnection.commit()}} directly.
# The server ({{IndexRegionObserver.preBatchMutate}}) rejects the batch with
{{StaleClusterRoleRecordException}}, because the cluster role is no longer
ACTIVE.
# {{executeMutation()}} throws the error again, unchanged.
{{FailoverPhoenixConnection.wrapActionDuringFailover()}} is not on the call
stack, so nothing calls {{HighAvailabilityGroup.refreshClusterRoleRecord()}}.
The other CRR refresh triggers do not help a write-only application:
* The interval refresh runs only before non-point-lookup queries.
* The poller runs only when the client CRR has no active role.
* The stale-CRR catch in {{TableResultIterator}} covers only scans.
These calls have the same gap:
* {{Statement.executeUpdate(String)}}
* {{Statement.execute()}} and {{PreparedStatement.execute()}}
* {{executeBatch()}}
* the autocommit commit after a query
h3. How to reproduce
{noformat}
Properties props = new Properties();
props.setProperty("AutoCommit", "true");
Connection conn = DriverManager.getConnection(haUrl, props);
PreparedStatement ps = conn.prepareStatement("UPSERT INTO T VALUES (?, ?)");
ps.setInt(1, 1); ps.setInt(2, 1);
ps.executeUpdate(); // OK
// Server side: change cluster1 ACTIVE -> ACTIVE_TO_STANDBY -> STANDBY
ps.setInt(1, 2); ps.setInt(2, 2);
ps.executeUpdate(); // fails with StaleClusterRoleRecordException, CRR not
refreshed
ps.executeUpdate(); // fails again the same way, every time
{noformat}
h3. Proposed fix
* Add {{HighAvailabilityUtil.toFailoverInProgressIfStale(PhoenixConnection,
SQLException)}}. It applies only to an HA connection ({{getHAGroup() != null}})
whose error chain has a {{StaleClusterRoleRecordException}}. It then:
** increments {{HA_STALE_CRR_DETECTED_COUNT}} (global and per HA group);
** calls {{refreshClusterRoleRecord(true)}}, which closes the connections when
the cluster is no longer active;
** returns {{FAILOVER_IN_PROGRESS}} with the original error as the root cause.
* For all other errors, the helper returns the error unchanged.
* Call the helper in {{PhoenixStatement}}: in the outer catch of
{{executeQuery()}} and {{executeMutation()}}, and in {{executeBatch()}}. In
{{executeBatch()}}, the cause of the {{BatchUpdateException}} becomes
{{FAILOVER_IN_PROGRESS}}. Skip the {{setAutoCommit()}} reset in its {{finally}}
block when the connection is closed, so the original error is not hidden.
* In {{FailoverPhoenixConnection.wrapActionDuringFailover()}}, throw
{{FAILOVER_IN_PROGRESS}} again unchanged. This prevents a second CRR refresh
and a double metric count.
* Make {{isStaleClusterRoleRecordExceptionExistsInThrowable()}} null-safe.
Non-HA connections do not change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)