Zhengcy05 commented on issue #8197:
URL:
https://github.com/apache/incubator-seata/issues/8197#issuecomment-5276818193
I looked into this issue, and the evidence points to a race between the
initial commit cleanup and the scheduled async-commit worker, rather than a
late branch registration.
For an AT transaction, `DefaultCore.commit()` changes the global status to
`AsyncCommitting` before `globalSession.clean()` has finished releasing all
file locks. The scheduled `handleAsyncCommitting()` task can observe that
status and call `doGlobalCommit(..., true)` without acquiring the same
`GlobalSession` lock.
The async commit path then removes committed `BranchSession` objects while
`FileLockManager.releaseGlobalSessionLock()` is iterating the same `ArrayList`,
which explains the `ConcurrentModificationException`.
This also explains the residual lock: while the status is `AsyncCommitting`,
`GlobalSession.unlockBranch()` assumes that the initial global cleanup has
already released the lock and skips the per-branch unlock. If that branch is
removed after the initial cleanup aborts, subsequent cleanup can no longer find
it, leaving the entry in `FileLocker.LOCK_MAP`.
I suggest serializing the scheduled async-commit handling with the same
`GlobalSession` lock and rechecking the status inside the lock. A snapshot-only
change in `FileLockManager` would avoid the iterator exception but would not
guarantee that a concurrently removed branch has been unlocked.
Below is my understanding of the timing of this bug。
```mermaid
sequenceDiagram
autonumber
participant A as Commit Thread A
participant GS as GlobalSession
participant BL as branchSessions (ArrayList)
participant B as Async Commit Thread B
A->>GS: Acquire GlobalSession lock
activate A
A->>GS: Change status to AsyncCommitting
Note over A,GS: Global lock cleanup has not finished
B->>GS: Scheduled task observes AsyncCommitting
activate B
Note over B,GS: Thread B does not acquire the GlobalSession lock
A->>BL: Iterate branches to release global locks
loop For each BranchSession
A->>BL: Read next branch
end
B->>B: Commit branch successfully
B->>BL: Remove committed BranchSession
BL--xA: Iterator detects concurrent modification
Note over A,BL: ConcurrentModificationException
deactivate A
Note over GS,BL: Cleanup stops before all global locks are released
deactivate B
```
Chinese Version
```mermaid
sequenceDiagram
autonumber
participant A as 提交线程 A
participant GS as GlobalSession
participant BL as branchSessions（ArrayList）
participant B as 异步提交线程 B
A->>GS: 获取 GlobalSession 锁
activate A
A->>GS: 将状态修改为 AsyncCommitting
Note over A,GS: 此时全局锁清理尚未完成
B->>GS: 定时任务发现 AsyncCommitting 状态
activate B
Note over B,GS: 线程 B 没有获取 GlobalSession 锁
A->>BL: 遍历分支并释放全局锁
loop 遍历每个 BranchSession
A->>BL: 读取下一个分支
end
B->>B: 分支提交成功
B->>BL: 删除已提交的 BranchSession
BL--xA: 迭代器检测到并发修改
Note over A,BL: 抛出 ConcurrentModificationException
deactivate A
Note over GS,BL: 清理被中断，部分全局锁未能释放
deactivate B
```
If this direction looks good, I’d like to work on it and add a concurrency
regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
