GitHub user WangzJi created a discussion: Seata Benchmark CLI - Design Discussion
**Status:** Draft **Related Issue:** [#7646](https://github.com/apache/incubator-seata/issues/7646) ## Table of Contents - [1. Background & Motivation](#1-background--motivation) - [2. Requirements](#2-requirements) - [3. User Interface Design](#3-user-interface-design) - [4. Technical Design](#4-technical-design) - [5. Open Questions](#5-open-questions) - [6. Implementation Roadmap](#6-implementation-roadmap) --- ## 1. Background & Motivation ### 1.1 Current Situation Seata currently lacks an official command-line benchmark tool for performance testing. Users who want to evaluate Seata's performance need to: - Write custom test code - Set up complex test environments - Manually collect and analyze metrics - Lack standardized benchmarking methodology This creates barriers for: - New users evaluating Seata performance - Performance regression testing in CI/CD - Comparing different transaction modes (AT/TCC/XA/SAGA) - Tuning Seata Server configuration ### 1.2 Goals Provide a **simple, standardized, command-line benchmark tool** that: 1. **Ease of Use**: Minimal parameters to run benchmarks 2. **Standardized Metrics**: Consistent output format (RT, QPS, success rate, latency percentiles) 3. **Multiple Modes**: Support AT and TCC transaction modes initially 4. **No Database Dependency**: Focus on empty transactions to test Seata Server capacity 5. **Extensible**: Design for future enhancements (more modes, real scenarios, distributed testing) --- ## 2. Requirements ### 2.1 Functional Requirements **Must Have (v1.0):** - Support **AT mode** transaction benchmarking - Support **TCC mode** transaction benchmarking - Accept minimal parameters: Seata Server address, mode, TPS, threads, duration - Output core metrics: Total transactions, success rate, average TPS, RT, P95/P99 latency - Empty transaction scenario (no business database required) **Should Have (v1.0):** - Warmup support to exclude initial ramp-up from statistics - Latency sampling to prevent OOM on large-scale tests - CSV export for post-analysis **Could Have (Future):** - Real-time TUI (Terminal User Interface) display - Real business scenarios (account transfer, order creation) - XA/SAGA mode support - Distributed benchmark coordination ### 2.2 Non-Functional Requirements - **Usability**: One-line command to run basic benchmark - **Performance**: Support high TPS testing (1000+ TPS) - **Portability**: Single executable JAR, JDK 8+ compatible - **Observability**: Clear progress indication and error reporting - **Extensibility**: Pluggable executor design for adding new modes --- ## 3. User Interface Design ### 3.1 Command-Line Interface **Basic Usage:** ```bash # AT mode benchmark java -jar seata-benchmark-cli.jar \ --server 127.0.0.1:8091 \ --mode AT \ --tps 100 \ --duration 60 # TCC mode benchmark java -jar seata-benchmark-cli.jar \ --server 127.0.0.1:8091 \ --mode TCC \ --tps 200 \ --threads 20 \ --duration 120 ``` **Advanced Options:** ```bash java -jar seata-benchmark-cli.jar \ --server 127.0.0.1:8091 \ --mode AT \ --tps 500 \ --threads 50 \ --duration 300 \ --warmup-duration 30 \ --export-csv results.csv \ --application-id benchmark-app \ --tx-service-group default_tx_group ``` ### 3.2 Parameters | Parameter | Short | Type | Default | Description | |-----------|-------|------|---------|-------------| | `--server` | `-s` | String | **Required** | Seata Server address (host:port) | | `--mode` | `-m` | String | **Required** | Transaction mode: AT, TCC | | `--tps` | `-t` | Integer | 100 | Target transactions per second | | `--threads` | | Integer | 10 | Number of concurrent threads | | `--duration` | `-d` | Integer | 60 | Benchmark duration in seconds | | `--warmup-duration` | | Integer | 0 | Warmup period in seconds (excluded from final stats) | | `--export-csv` | | String | - | Export metrics to CSV file | | `--application-id` | | String | benchmark-app | Seata application ID | | `--tx-service-group` | | String | default_tx_group | Seata transaction service group | | `--help` | `-h` | - | - | Show help message | | `--version` | `-V` | - | - | Show version information | ### 3.3 Output Format **Console Output (Real-time Progress):** ``` [00:05] 500 txns, 100.2 txns/sec, 98.0% success, avg 12ms [00:10] 1000 txns, 100.1 txns/sec, 99.0% success, avg 11ms [00:15] 1500 txns, 99.8 txns/sec, 99.2% success, avg 13ms ... ``` **Final Report:** ``` =================================================== Seata Benchmark Final Report (AT Mode) =================================================== Total Transactions: 6,000 Success Count: 5,940 Failed Count: 60 Success Rate: 99.00% Average TPS: 100.2 Elapsed Time: 60 seconds Latency Statistics: P50: 12 ms P95: 45 ms P99: 89 ms Max: 230 ms Warmup Statistics (excluded): Warmup Transactions: 500 Warmup Duration: 5 seconds =================================================== ``` **CSV Export Format:** ```csv Metric,Value Total Transactions,6000 Success Count,5940 Failed Count,60 Success Rate (%),99.00 Average TPS,100.2 Elapsed Time (s),60 Latency P50 (ms),12 Latency P95 (ms),45 Latency P99 (ms),89 Latency Max (ms),230 Warmup Transactions,500 Export Timestamp,2025-12-01T10:30:45Z ``` --- ## 4. Technical Design ### 4.1 Architecture ``` ┌─────────────────────────────────────────────┐ │ BenchmarkApplication (CLI) │ │ (PicoCLI Framework) │ └──────────────────┬──────────────────────────┘ │ ┌──────────┴──────────┐ │ │ ┌────▼─────┐ ┌──────▼──────┐ │ Config │ │ Seata TM │ │ Loader │ │ Client │ └────┬─────┘ └──────┬──────┘ │ │ └──────────┬──────────┘ │ ┌──────────▼──────────────┐ │ TransactionExecutor │ │ (Strategy Pattern) │ ├─────────────────────────┤ │ - ATModeExecutor │ │ - TCCModeExecutor │ └──────────┬──────────────┘ │ ┌──────────▼──────────────┐ │ WorkloadGenerator │ │ (Guava RateLimiter) │ └──────────┬──────────────┘ │ ┌──────────▼──────────────┐ │ MetricsCollector │ │ (Latency Sampling) │ └──────────┬──────────────┘ │ ┌──────────▼──────────────┐ │ Reporter │ │ (Console + CSV) │ └─────────────────────────┘ ``` ### 4.2 Core Components #### 4.2.1 BenchmarkApplication (Main Entry) - CLI framework: **PicoCLI** (annotation-driven, modern) - Responsibilities: - Parse command-line arguments - Initialize Seata TM client - Orchestrate benchmark execution - Handle graceful shutdown #### 4.2.2 TransactionExecutor (Strategy Pattern) **Interface:** ```java public interface TransactionExecutor { void init(BenchmarkConfig config); TransactionRecord execute(); void destroy(); } ``` **Implementations:** - `ATModeExecutor`: Empty AT transaction (begin -> commit, no SQL) - `TCCModeExecutor`: Mock TCC transaction (empty try/confirm/cancel) #### 4.2.3 WorkloadGenerator - TPS control: **Guava RateLimiter** - Multi-threaded execution: Fixed thread pool - Graceful termination on duration timeout - Pause/resume support (optional) #### 4.2.4 MetricsCollector **Key Metrics:** - Total transactions (counter) - Success/failure counts (atomic counters) - Latency tracking with **sampling** (inspired by Kafka) - Sample every Nth transaction to limit memory - Max 500,000 samples stored - Current and average TPS calculation - Percentile calculation: P50, P95, P99 **Sampling Strategy:** ```java int sampling = totalTransactions / Math.min(totalTransactions, 500_000); if (currentTransaction % sampling == 0) { storeLatency(latency); } ``` #### 4.2.5 Reporter - Console progress updates (window-based, every 5 seconds) - Final report formatter - CSV exporter (optional) ### 4.3 Empty Transaction Implementation #### AT Mode Empty Transaction ```java public TransactionRecord execute() { GlobalTransaction tx = GlobalTransactionContext.getCurrentOrCreate(); long startTime = System.currentTimeMillis(); try { tx.begin(30000, "benchmark-at-tx"); String xid = tx.getXid(); // No SQL operations - pure Seata protocol overhead tx.commit(); long duration = System.currentTimeMillis() - startTime; return new TransactionRecord(xid, "Committed", duration, true); } catch (TransactionException e) { tx.rollback(); long duration = System.currentTimeMillis() - startTime; return new TransactionRecord(null, "Failed", duration, false); } } ``` #### TCC Mode Mock Transaction ```java @LocalTCC public interface BenchmarkTCCAction { @TwoPhaseBusinessAction(name = "benchmarkAction", commitMethod = "commit", rollbackMethod = "rollback") boolean prepare(BusinessActionContext context); boolean commit(BusinessActionContext context); boolean rollback(BusinessActionContext context); } // Implementation: All methods return true (no-op) public class BenchmarkTCCActionImpl implements BenchmarkTCCAction { public boolean prepare(BusinessActionContext context) { return true; // Mock success } public boolean commit(BusinessActionContext context) { return true; // Mock success } public boolean rollback(BusinessActionContext context) { return true; // Mock success } } ``` ### 4.4 Technology Stack | Component | Technology | Rationale | |-----------|-----------|-----------| | CLI Framework | PicoCLI 4.7.5 | Modern, annotation-based, extensive validation | | Rate Limiting | Guava RateLimiter | Proven, accurate TPS control | | Seata Client | seata-all (latest) | Official Seata SDK | | Logging | SLF4J + Logback | Standard Java logging | | Build Tool | Maven | Consistent with Seata project | | Packaging | Maven Shade Plugin | Single executable JAR | --- ## 5. Open Questions Please provide your feedback on the following design decisions: ### Q1: TUI (Terminal User Interface) Support? **Options:** - **A:** Plain text output only (like Kafka - simple, scriptable) - **B:** TUI with real-time updates (like htop - visual, intuitive) - **C:** Both supported via `--tui` flag (flexible, more complexity) **Recommendation:** Start with A (plain text), add B in v2.0 if community requests ### Q2: TCC Mode Implementation? **Options:** - **A:** Mock TCC (empty try/confirm/cancel, no database) - **B:** Real TCC scenario (account freeze/unfreeze, requires database) - **C:** Start with A, add B as `--scenario` option in future **Recommendation:** A for v1.0 (aligns with "no database dependency" goal) ### Q3: Monitoring global_table/branch_table? **Options:** - **A:** No monitoring (avoids exposing Seata Server DB credentials) - **B:** Optional monitoring via `--monitor-db` parameter - **C:** AT mode only (TCC has no branch_table to monitor) **Recommendation:** A for v1.0 (keep it simple) ### Q4: Result Export Formats? **Options:** - **A:** Console output only - **B:** CSV export via `--export-csv` - **C:** JSON export via `--export-json` - **D:** B + C (multiple formats) **Recommendation:** B (CSV is sufficient for most analysis needs) ### Q5: CLI Framework Choice? **Options:** - **A:** PicoCLI (modern, annotation-based, good validation) - **B:** argparse4j (Kafka's choice, Python-style) - **C:** joptsimple (lightweight, Java-native) **Recommendation:** A (aligns with modern Java practices) ### Q6: XA Mode Support? **Options:** - **A:** Not in v1.0 (focus on AT/TCC as per community meeting) - **B:** Support AT/TCC/XA in v1.0 **Recommendation:** A (community meeting prioritized AT/TCC) ### Q7: Warmup Support? **Options:** - **A:** Must have in v1.0 (Kafka best practice) - **B:** Optional in v1.0 via `--warmup-duration` - **C:** Defer to v2.0 **Recommendation:** B (important for accurate benchmarks, but optional parameter) ### Q8: Latency Percentiles? **Options:** - **A:** P50, P95, P99 only - **B:** Add P99.9 (important for tail latency analysis) - **C:** Configurable percentiles via parameter **Recommendation:** B (P99.9 is valuable with minimal cost) --- ## 6. Implementation Roadmap ### Phase 1: MVP (v0.1) **Scope:** - AT mode empty transaction executor - TCC mode mock transaction executor - Basic CLI (server, mode, tps, threads, duration) - Plain text progress and final report - Core metrics: total txns, success rate, avg TPS, P50/P95/P99 latency **Deliverables:** - Executable JAR - README with usage examples - Basic test coverage ### Phase 2: Enhancement (v0.2) **Scope:** - Latency sampling (prevent OOM) - Warmup support (`--warmup-duration`) - CSV export (`--export-csv`) - P99.9 percentile - Window-based progress reporting - Comprehensive error handling **Deliverables:** - Enhanced JAR with all features - User guide documentation - Performance test report (validate tool can handle 1000+ TPS) ### Phase 3: Advanced Features (v1.0) - TBD **Scope (based on community feedback):** - TUI real-time display (optional) - Real business scenarios (configurable via `--scenario`) - XA/SAGA mode support - Monitoring integration (global_table/branch_table) - Distributed benchmark coordination **Deliverables:** - Production-ready v1.0 release - Complete documentation --- ## 7. Community Feedback Please share your thoughts on: 1. **Scope:** Is the v1.0 scope appropriate? Too ambitious or too limited? 2. **Parameters:** Are the proposed CLI parameters sufficient? Any missing? 3. **Output:** Is the console + CSV output adequate, or should we prioritize TUI? 4. **Open Questions:** Your preferences on Q1-Q8 above 5. **Roadmap:** Any changes to the proposed timeline? **How to Provide Feedback:** - Comment on this discussion thread - Email the dev mailing list - Join the community meeting --- **Thank you for your valuable input!** GitHub link: https://github.com/apache/incubator-seata/discussions/7824 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
