[
https://issues.apache.org/jira/browse/GOBBLIN-2096?focusedWorklogId=923982&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923982
]
ASF GitHub Bot logged work on GOBBLIN-2096:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 21/Jun/24 14:23
Start Date: 21/Jun/24 14:23
Worklog Time Spent: 10m
Work Description: umustafi commented on code in PR #3982:
URL: https://github.com/apache/gobblin/pull/3982#discussion_r1645227282
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/MysqlMultiActiveLeaseArbiter.java:
##########
@@ -420,15 +428,25 @@ protected GetEventInfoResult
createGetInfoResult(ResultSet resultSet) throws IOE
* near past for it.
* @return int corresponding to number of rows updated by INSERT statement
to acquire lease
*/
- protected int attemptLeaseIfNewRow(DagActionStore.DagAction dagAction)
throws IOException {
+ protected int attemptLeaseIfNewRow(DagActionStore.DagAction dagAction,
ExponentialBackoff exponentialBackoff) throws IOException {
String formattedAcquireLeaseNewRowStatement =
String.format(ACQUIRE_LEASE_IF_NEW_ROW_STATEMENT,
this.leaseArbiterTableName);
return
dbStatementExecutor.withPreparedStatement(formattedAcquireLeaseNewRowStatement,
insertStatement -> {
completeInsertPreparedStatement(insertStatement, dagAction);
try {
return insertStatement.executeUpdate();
- } catch (SQLIntegrityConstraintViolationException e) {
+ } catch (SQLTransientException e) {
+ try {
+ if (exponentialBackoff.awaitNextRetryIfAvailable()) {
+ return attemptLeaseIfNewRow(dagAction, exponentialBackoff);
+ }
+ } catch (InterruptedException | IOException e2) {
+ throw new IOException(e2);
+ }
+ return 0;
Review Comment:
Oh good call, doing that
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/MysqlMultiActiveLeaseArbiter.java:
##########
@@ -254,7 +256,8 @@ public LeaseAttemptStatus
tryAcquireLease(DagActionStore.DagActionLeaseObject da
log.debug("tryAcquireLease for [{}, is; {}, eventTimestamp: {}] - CASE
1: no existing row for this dag action,"
+ " then go ahead and insert", dagActionLeaseObject.getDagAction(),
dagActionLeaseObject.isReminder() ? "reminder" : "original",
dagActionLeaseObject.getEventTimeMillis());
- int numRowsUpdated =
attemptLeaseIfNewRow(dagActionLeaseObject.getDagAction());
+ int numRowsUpdated =
attemptLeaseIfNewRow(dagActionLeaseObject.getDagAction(),
+
ExponentialBackoff.builder().maxRetries(3).initialDelay(10L).build());
Review Comment:
increased to be in range 20,220
Issue Time Tracking
-------------------
Worklog Id: (was: 923982)
Time Spent: 1h (was: 50m)
> retry Transient SQL Exception for MALA
> ---------------------------------------
>
> Key: GOBBLIN-2096
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2096
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We're seeing a MySQLTransactionRollbackException: Deadlock found when trying
> to get lock when calling MysqlMultiActiveLeaseArbiter. attemptLeaseIfNewRow()
> in the rare case when two hosts check the lease arbiter table at the same
> time, see there's no entry matching the current primary key and try to insert
> it at the exact same time (both wait for lock on the same row). Normally,
> there's slight discrepancy that results in an
> IntegrityConstraintViolationException if one insert succeeds before the other
> OR one finishes the write early and the other sees the entry when reading the
> getResult of the arbiter table. This PR adds retries with exponential backoff
> for transient SQL exceptions to the insert statement. Note, it's not needed
> for other cases when the INSERT/UPDATE statements are conditional on a read
> value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
