LegendPei opened a new pull request, #8208:
URL: https://github.com/apache/incubator-seata/pull/8208
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- Please make sure you have read and understood the contributing
guidelines -->
- [ ] I have read the
[CONTRIBUTING.md](https://github.com/apache/incubator-seata/blob/2.x/CONTRIBUTING.md)
guidelines.
- [ ] I have registered the PR
[changes](https://github.com/apache/incubator-seata/tree/2.x/changes).
### Ⅰ. Describe what this PR did
This PR continues to optimize RocksDB FileMode in #8149, focusing on
handling the upper bound of background workload, Seata lifecycle performance,
WAL/recovery semantics, benchmark evidence, and protected maintenance processes.
The main content is as follows:
1. Extend the existing diagnostics, checkpoint, and benchmark harnesses in
#8149. Add independent JVM repeat, A/B sequence control, real payload,
scan/read/delete counts, p50/p95/p99, SST/WAL/compaction, and foreground
interference models to the benchmark.
2. Introduce `none` and `periodic` WAL sync strategies, and supplement with
clean-shutdown marker, strict sync before shutdown, exception-safe resource
release, scheduling shutdown race handling, and process-crash recovery harness.
3. Change status/begin/full scan and Coordinator retry/end-state queries to
protected scans with limit, deadline, cursor, and scan stats; use k-way merge
with resumable scanning for multi-state queries to avoid long-term occupation
of the first page by failed sessions.
4. Change the full cleanup of orphan locks during the startup phase to a
background task with persistent cursors, batch size limits, inter-round rate
limits, and interruption recovery, while retaining the conservative cleanup
semantics after a crash.
5. Exposed block cache, DB/CF write buffer, `maxTotalWalSize`, compaction,
and WAL-related parameters, but did not upgrade profiles that failed the
benchmark gate to their default values.
6. Optimize `GLOBAL_REMOVE`, branch/lock cleanup, and lock release within
the lifecycle boundary of xid. Use RangeDelete within a provably safe and
ordered index range, and continue to use a single `WriteBatch` to maintain
atomicity of state and index updates.
7. Add PAGE/FULL verify, repair plan, dry-run, quota execution, checkpoint
pre-protection, progress persistence, and fault recovery to the #8149 basic
verify and snapshot interfaces. High-cost maintenance actions will not enter
the normal transaction hot path.
Main benchmark results:
Machine parameters:
| Item | Value |
|---|---|
| OS | Ubuntu 24.04.2 LTS，Linux 6.8 |
| CPU | QEMU/KVM，12 vCPU |
| Memory | 15GiB |
| Disk | Virtual mechanical disk/shared storage |
| JDK | OpenJDK 21.0.11 |
| Maven | 3.9.6/3.9.x |
| RocksDB JNI | 10.10.1.1 |
| Common JVM | `-Xms2g -Xmx4g`；a small number of early matrices use `-Xms1g
-Xmx3g` |
| Scenario | Baseline | Phase 4 Candidate | Result |
|---|---:|---:|---:|
| cleanup `global_remove_with_branches`，1M×2×2，triplicated | 16,455.966
ops/s | 25,245.184 ops/s | 吞吐 `+53.4%` |
| write `global_remove`，1M×2×2，triplicated | 14,929.260 ops/s，p99 0.159ms |
23,407.043 ops/s，p99 0.105ms | Throughput `+56.8%`，p99 `-34.0%` |
| `lock.release_branch`，1M×2×2，triplicated | 13,047.133 ops/s，p99 0.152ms |
18,402.559 ops/s，p99 0.112ms | Throughput `+41.0%`，p99 `-26.3%` |
| `lock.release_global`，1M×2×2，triplicated | 13,640.268 ops/s，p99 0.143ms |
19,869.311 ops/s，p99 0.103ms | Throughput `+45.7%`，p99 `-28.0%` |
| status bounded scan，100K→200K，total overdue fixed at 1,000 | 1,001 rows
scanned | 1,001 rows scanned | Single scan volume remains bounded when total
data doubles |
| high fan-out `GLOBAL_REMOVE`，50 branch×10 lock | 318.951 ops/s，p99 4.951ms
| 416.840 ops/s，p99 3.636ms | Throughput`+30.7%`，p99 `-26.6%` |
The performance improvement mentioned here is derived from repeated tests
conducted on the same machine, with the same scale, workload, and independent
JVM. Since bounded scan and full scan yield different return sizes, it is only
used to demonstrate the upper bound of a single workload, without calculating
the multiple improvement.
### Ⅱ. Does this pull request fix one issue?
<!-- If that, add "fixes #xxx" below in the next line, for example, fixes
#97. -->
fixes#8114
### Ⅲ. Why don't you add test cases (unit test/integration test)?
Test coverage:
- RocksDB parameter parsing, illegal value verification, DB/CF options
application, and diagnostic snapshot.
- WAL periodic sync, shutdown strict sync, clean marker, sync failure,
executor close race, and crash recovery harness.
- status/begin/multi-status cursor, deadline, bounded scan stats,
Coordinator round-robin fairness, and stale index.
- Equivalent semantics of RangeDelete and scan-delete, duplicate deletion,
restart recovery, branch/lock fan-out, and atomic `WriteBatch`.
- Persistence of cursor, rate limiting, restart continuation, exception
recovery, and foreground interference for orphan lock background cleanup.
- checkpoint, PAGE/FULL verify, repair dry-run, guard, fault injection,
progress recovery, and Raft snapshot.
- Benchmark parameter parsing, CSV header, repeat summary, actual
rows/pointReads, fixed expiration time, and probe startup synchronization.
### Ⅳ. Describe how to verify it
### Ⅴ. Special notes for reviews
This PR is a follow-up optimization of #8149 and depends on #8149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
