Lokesh Khurana created PHOENIX-8068:
---------------------------------------

             Summary: HA client: autocommit statement DML does not refresh a 
stale cluster role record
                 Key: PHOENIX-8068
                 URL: https://issues.apache.org/jira/browse/PHOENIX-8068
             Project: Phoenix
          Issue Type: Sub-task
            Reporter: Lokesh Khurana
            Assignee: Lokesh Khurana


h3. Problem

With autocommit on, statement-level DML on an HA connection 
({{FailoverPhoenixConnection}}) does not fail over after a consistent failover. 
The server rejects each write with {{StaleClusterRoleRecordException}}. The 
client does not refresh its cluster role record (CRR), so it stays on the old 
active cluster. Every later write fails the same way, in a few milliseconds, 
until the application opens a new connection.

The application does not get {{FAILOVER_IN_PROGRESS}} (error code 1990), so it 
has no signal to call {{FailoverPhoenixConnection.failover()}} or to reconnect.

An explicit {{conn.commit()}} does not have this problem. That call goes 
through {{FailoverPhoenixConnection}}, which maps the error and refreshes the 
CRR.

h3. Code path

# {{FailoverPhoenixConnection.prepareStatement()}} and {{createStatement()}} 
wrap only the create step. They return a raw {{PhoenixStatement}} on the 
wrapped {{PhoenixConnection}}.
# {{PhoenixPreparedStatement.executeUpdate()}} calls 
{{PhoenixStatement.executeMutation()}}. With autocommit on, it calls 
{{PhoenixConnection.commit()}} directly.
# The server ({{IndexRegionObserver.preBatchMutate}}) rejects the batch with 
{{StaleClusterRoleRecordException}}, because the cluster role is no longer 
ACTIVE.
# {{executeMutation()}} throws the error again, unchanged. 
{{FailoverPhoenixConnection.wrapActionDuringFailover()}} is not on the call 
stack, so nothing calls {{HighAvailabilityGroup.refreshClusterRoleRecord()}}.

The other CRR refresh triggers do not help a write-only application:
* The interval refresh runs only before non-point-lookup queries.
* The poller runs only when the client CRR has no active role.
* The stale-CRR catch in {{TableResultIterator}} covers only scans.

These calls have the same gap:
* {{Statement.executeUpdate(String)}}
* {{Statement.execute()}} and {{PreparedStatement.execute()}}
* {{executeBatch()}}
* the autocommit commit after a query

h3. How to reproduce

{noformat}
Properties props = new Properties();
props.setProperty("AutoCommit", "true");
Connection conn = DriverManager.getConnection(haUrl, props);
PreparedStatement ps = conn.prepareStatement("UPSERT INTO T VALUES (?, ?)");
ps.setInt(1, 1); ps.setInt(2, 1);
ps.executeUpdate();   // OK

// Server side: change cluster1 ACTIVE -> ACTIVE_TO_STANDBY -> STANDBY

ps.setInt(1, 2); ps.setInt(2, 2);
ps.executeUpdate();   // fails with StaleClusterRoleRecordException, CRR not 
refreshed
ps.executeUpdate();   // fails again the same way, every time
{noformat}

h3. Proposed fix

* Add {{HighAvailabilityUtil.toFailoverInProgressIfStale(PhoenixConnection, 
SQLException)}}. It applies only to an HA connection ({{getHAGroup() != null}}) 
whose error chain has a {{StaleClusterRoleRecordException}}. It then:
** increments {{HA_STALE_CRR_DETECTED_COUNT}} (global and per HA group);
** calls {{refreshClusterRoleRecord(true)}}, which closes the connections when 
the cluster is no longer active;
** returns {{FAILOVER_IN_PROGRESS}} with the original error as the root cause.
* For all other errors, the helper returns the error unchanged.
* Call the helper in {{PhoenixStatement}}: in the outer catch of 
{{executeQuery()}} and {{executeMutation()}}, and in {{executeBatch()}}. In 
{{executeBatch()}}, the cause of the {{BatchUpdateException}} becomes 
{{FAILOVER_IN_PROGRESS}}. Skip the {{setAutoCommit()}} reset in its {{finally}} 
block when the connection is closed, so the original error is not hidden.
* In {{FailoverPhoenixConnection.wrapActionDuringFailover()}}, throw 
{{FAILOVER_IN_PROGRESS}} again unchanged. This prevents a second CRR refresh 
and a double metric count.
* Make {{isStaleClusterRoleRecordExceptionExistsInThrowable()}} null-safe.

Non-HA connections do not change.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to