makssent commented on issue #36972: URL: https://github.com/apache/shardingsphere/issues/36972#issuecomment-3664156488
I think I’ve managed to figure out the issue. It turned out to be related to the VM configuration, specifically the **virtio drivers**. In our setup, they were effectively overloading a single CPU core and preventing the proxy from fully utilizing available resources, which limited performance. After moving **ShardingSphere Proxy** into a **Docker container**, the problem disappeared and scaling started to work as expected. At peak, we observed a performance improvement of about **x4.1–x4.2**. Further limitations were no longer environment-related but came from the proxy itself becoming a bottleneck: - on **4 nodes**, the QPS gain was noticeably smaller than between 2 and 3 nodes; - on **5 nodes**, there was almost no improvement — the proxy stopped scaling. That said, scalability was clearly confirmed up to **3–4 nodes**. --- ### Test results | **Configuration** | **QPS #1** | **QPS #2** | **QPS #3** | **Avg QPS** | **Avg p95 (ms)** | **Gain vs single (40M)** | |-----------------------------------------------|------------|------------|------------|-------------|------------------|--------------------------| | **Single JDBC (no Proxy, 40M records)** | 1919.38 | 1819.93 | 2023.55 | **1920.95** | **129.62** | — | | ShardingSphere Proxy (RDB, 2 nodes, 8M each) | 5162.60 | 5000.10 | 5265.58 | **5142.76** | 29.99 | +167.8% (×2.68) | | ShardingSphere Proxy (RDB, 3 nodes, 8M each) | 7080.13 | 7237.47 | 7239.47 | **7185.69** | 33.06 | +274.1% (×3.74) | | ShardingSphere Proxy (RDB, 4 nodes, 8M each) | 8707.05 | 8430.93 | 8808.70 | **8648.89** | 33.25 | +350.3% (×4.50) | | **ShardingSphere Proxy (RDB, 5 nodes, 8M each)** | 8862.07 | 7492.58 | 7617.75 | **7990.80** | 63.96 | +316.1% (×4.16) | --- ### About result reliability Some of these numbers may look **too optimistic** (for example, around **x2.5 on 2 nodes**). This is mainly due to **unstable VM performance** rather than ShardingSphere itself. We observed cases where: - the single-node setup produced about **2500 QPS**, - and an hour later, under otherwise identical conditions, only **~1800 QPS**. In addition, the comparison was not fully symmetric: - the cluster was tested with **16–24M records**, - while the single setup used **40M records**. To obtain truly accurate numbers, such tests should be performed on **bare-metal hardware**. However, our goal at this stage was simply to confirm **whether scalability exists at all**, and that was successfully demonstrated. We also observed real connection parallelism (around **200 connections per node**), which was sufficient for us to move forward. --- <details> <summary><b>About Firebird ReadOnly mode</b></summary> Initially, I was unable to get good performance on **Firebird** due to the **ReadOnly mode**. ReadOnly was enabled on the **single-node setup**, but **Firebird does not support ReadOnly mode in a clustered configuration**. Because of this, the comparison was incorrect and the performance gain looked very small. After disabling ReadOnly for the single-node setup as well, performance appeared and scalability became clearly visible. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
