[
https://issues.apache.org/jira/browse/GOBBLIN-2096?focusedWorklogId=923972&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-923972
]
ASF GitHub Bot logged work on GOBBLIN-2096:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 21/Jun/24 14:22
Start Date: 21/Jun/24 14:22
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3982:
URL: https://github.com/apache/gobblin/pull/3982#discussion_r1645196095
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/MysqlMultiActiveLeaseArbiter.java:
##########
@@ -254,7 +256,8 @@ public LeaseAttemptStatus
tryAcquireLease(DagActionStore.DagActionLeaseObject da
log.debug("tryAcquireLease for [{}, is; {}, eventTimestamp: {}] - CASE
1: no existing row for this dag action,"
+ " then go ahead and insert", dagActionLeaseObject.getDagAction(),
dagActionLeaseObject.isReminder() ? "reminder" : "original",
dagActionLeaseObject.getEventTimeMillis());
- int numRowsUpdated =
attemptLeaseIfNewRow(dagActionLeaseObject.getDagAction());
+ int numRowsUpdated =
attemptLeaseIfNewRow(dagActionLeaseObject.getDagAction(),
+
ExponentialBackoff.builder().maxRetries(3).initialDelay(10L).build());
Review Comment:
create it one time statically outside the method.
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/MysqlMultiActiveLeaseArbiterTest.java:
##########
@@ -201,9 +206,11 @@ Tests attemptLeaseIfNewRow() method to ensure a new row is
inserted if no row ma
@Test
public void testAcquireLeaseIfNewRow() throws IOException {
// Inserting the first time should update 1 row
-
Assert.assertEquals(mysqlMultiActiveLeaseArbiter.attemptLeaseIfNewRow(resumeDagAction),
1);
+
Assert.assertEquals(mysqlMultiActiveLeaseArbiter.attemptLeaseIfNewRow(resumeDagAction,
+ ExponentialBackoff.builder().maxRetries(3).initialDelay(10L).build()),
1);
Review Comment:
same here, though it is less impactful in tests
Issue Time Tracking
-------------------
Worklog Id: (was: 923972)
Time Spent: 20m (was: 10m)
> retry Transient SQL Exception for MALA
> ---------------------------------------
>
> Key: GOBBLIN-2096
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2096
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We're seeing a MySQLTransactionRollbackException: Deadlock found when trying
> to get lock when calling MysqlMultiActiveLeaseArbiter. attemptLeaseIfNewRow()
> in the rare case when two hosts check the lease arbiter table at the same
> time, see there's no entry matching the current primary key and try to insert
> it at the exact same time (both wait for lock on the same row). Normally,
> there's slight discrepancy that results in an
> IntegrityConstraintViolationException if one insert succeeds before the other
> OR one finishes the write early and the other sees the entry when reading the
> getResult of the arbiter table. This PR adds retries with exponential backoff
> for transient SQL exceptions to the insert statement. Note, it's not needed
> for other cases when the INSERT/UPDATE statements are conditional on a read
> value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
