This is an automated email from the ASF dual-hosted git repository.
menghaoran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/shardingsphere.git
The following commit(s) were added to refs/heads/master by this push:
new 3a617a9 Stress testing blog (#15158)
3a617a9 is described below
commit 3a617a9710a8f7ad74d06f5912b77257891e8fd7
Author: Yacine Si Tayeb <[email protected]>
AuthorDate: Fri Jan 28 22:30:20 2022 +0800
Stress testing blog (#15158)
---
...ingSphere_Shadow_Database_Feature_Upgrade.en.md | 238 +++++++++++++++++++++
.../img/Full_Link_Online_Stress_Testing_img_1.png | Bin 0 -> 7747 bytes
.../img/Full_Link_Online_Stress_Testing_img_10.png | Bin 0 -> 19383 bytes
.../img/Full_Link_Online_Stress_Testing_img_2.png | Bin 0 -> 29778 bytes
.../img/Full_Link_Online_Stress_Testing_img_3.jpeg | Bin 0 -> 59470 bytes
.../img/Full_Link_Online_Stress_Testing_img_4.png | Bin 0 -> 52649 bytes
.../img/Full_Link_Online_Stress_Testing_img_5.jpg | Bin 0 -> 87875 bytes
.../img/Full_Link_Online_Stress_Testing_img_6.png | Bin 0 -> 12014 bytes
.../img/Full_Link_Online_Stress_Testing_img_7.png | Bin 0 -> 30466 bytes
.../img/Full_Link_Online_Stress_Testing_img_8.png | Bin 0 -> 20625 bytes
.../img/Full_Link_Online_Stress_Testing_img_9.png | Bin 0 -> 23255 bytes
docs/blog/static/img/Hou_Yang_Photo.png | Bin 0 -> 94106 bytes
12 files changed, 238 insertions(+)
diff --git
a/docs/blog/content/material/Jan_28_Full_Link_Online_Stress_Testing_for_Production_Database_Apache_ShardingSphere_Shadow_Database_Feature_Upgrade.en.md
b/docs/blog/content/material/Jan_28_Full_Link_Online_Stress_Testing_for_Production_Database_Apache_ShardingSphere_Shadow_Database_Feature_Upgrade.en.md
new file mode 100644
index 0000000..f6c5c1e
--- /dev/null
+++
b/docs/blog/content/material/Jan_28_Full_Link_Online_Stress_Testing_for_Production_Database_Apache_ShardingSphere_Shadow_Database_Feature_Upgrade.en.md
@@ -0,0 +1,238 @@
+# Full Link Online Stress Testing for Production Database: Apache
ShardingSphere Shadow Database Feature Upgrade
+
+## What is full link stress testing?
+
+As the Internet industry is growing rapidly, businesses that operate with
large amounts of data are seeing a rapid expansion.
+
+Predictably, ever changing customer demand is having a significant impact on
the stability of their whole systems. For example, online food delivery
platforms receive most of their customer orders at noon and in the evening.
Online shopping sprees and time-limited sales promotions are also good examples.
+
+All businesses are served by a series of business systems which are
distributedly deployed in different machines. “Data Plannig” cannot only ensure
systems’ stability but also save costs, which are some of the major problems
for technology teams.
+
+To precisely get the right service from a specific machine, stress testing
should be conducted in the production environment. This can ensure the
authenticity of the environment and the data, significantly improving the
precision of “data planning”.
+
+## Shadow Database and Full-Link Stress Testing
+
+Performing a stress test on an online business system is evidently risky. For
example, data corruption or performance problems may arise.
+
+Just imagine how it would compromise customer experience, if customers found
their order has been lost or an unpaid order popped up?
+
+Full-link online stress testing implies a significant amount of complicated
work which requires cooperation between microservices and middlewares. [Apache
ShardingSphere](https://shardingsphere.apache.org/) focuses on database-level
solutions in full-link stress testing.
+
+With ShardingSphere’s powerful SQL parsing capability, Apache ShardingSphere
released the shadow database stress testing feature to determine shadow
databases by executing SQL, and to meet the online stress testing needs in
complicated business scenarios with the flexible configuration of shadow
algorithms. By routing stress testing traffic to shadow databases, and normal
online traffic to production database, stress testing data will be isolated and
the data corruption problem is solved.
+
+## Shadow Database Function Upgrading
+
+Shadow database function was initially realized in version 4.1.0 by adding a
logic shadow column. By parsing, executing, routing and rewriting SQL, Apache
ShardingSphere deleted the shadow column and column value. Users do not need to
setup or do anything during this process. They only need to modify SQL
accordingly, add shadow fields and corresponding configurations.
+
+### Two pain points occur when adding shadow column:
+
+1. Before performing stress testings, users need to modify test-related SQL
according to actual business needs.
+2. SQL modification will increase implementation damage and reduce the
accuracy of stress testing results. After discussion in the ShardingSphere
community, we decided to upgrade shadow database functions. Apache
ShardingSphere 4.1.1 GA shadow database API had relatively simple
functionaloity. Whether to open shadow database is determined by corresponding
values of the logicColumn.
+
+
+
+The upgraded shadow database API of 5.0.0 GA is more powerful. Users can
determine whether the shadow database function is enabled through the “enable”
attribute. Configurable shadow table can determine what needs to be stress
tested by means of a table, and supports multiple shadow algorithms. For
example, column value matching algorithm, regular expression matching and SQL
comment matching algorithm.
+
+
+
+## Shadow Database in Practice
+
+Online full-link stress testing diagram:
+
+
+
+### Prepare stress testing environment:
+Suppose an e-commerce website needs to perform online stress testing for an
order (demonstrating how to use stand-alone deployment). Suppose stress testing
table `t_order` is an order table, and the ID of the test user is 0.
+
+The data generated by test user order is executed on ds_shadow shadow
database, and the production data is executed on ds production database.
+
+**Prepare testing environment:**
+
+1. Download ShardingSphere-Proxy 5.0.0 GA from the [Download
Page](https://shardingsphere.apache.org/document/5.0.0/en/downloads/), and for
installation configuration details, please refer to
[ShardingSphere-Proxy-Quick-Start](https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/).
+2. Configure ShardingSphere-Proxy in the hypothetical stress testing scenario
mentioned above:
+
+**server.yaml**
+
+
+
+**config-shadow.yaml**
+
+
+
+3. Order service
+
+Order-related businesses are not discussed here. Considering the simplest
request taking and order table inserting as an example, the order table
structure is as follows:
+
+- Order Table Structure
+
+
+
+### Stress Testing Process Simulation
+- Use `postman` to simulate the order created by test user, which is as
follows:
+
+
+
+- SQL executor being routed to and executed in shadow database can be seen in
ShardingSphere-Proxy execution log:
+
+
+
+### Verify Stress Testing Results :
+
+- Shadow database `ds_shadow` executes query sentence `SELECT * FROM t_order;`
+
+Query result:
+
+
+
+- Production database ds executes query sentence `SELECT * FROM t_order;`
+
+Query result:
+
+
+
+The data generated from test user order creation will be routed to shadow
database. For more complicated configurations, please refer to Shadow Database
Stress Testing in the [ShardingSphere official
document](https://shardingsphere.apache.org/document/5.0.0/cn/features/shadow/).
+
+### Complete Solution for Full-Link Online Stress Testing — CyborgFlow
+
+As mentioned in the introduction, full-link online stress testing is a
complicated task that requires collaboration between microservices and
middlewares to meet the needs of different traffic and stress testing tag
transmissions.
+
+Additonally, the testing service should be stateless and immediately
available. [CyborgFlow](https://github.com/SphereEx/CyborgFlow), which is
jointly maintained by Apache ShardingSphere, Apache APISIX and Apache
SkyWalking provides out-of-the-box (OoTB) solution to run load test in your
online system.
+
+[Apache APISIX](https://apisix.apache.org/) is responsible for making tags on
testing data at the gateway layer, while [Apache
SkyWalking](https://skywalking.apache.org/) is responsible for transmission
through the whole scheduling link, and finally, Apache ShardingSphere-Proxy
will isolate data and route testing data to the shadow database.
+
+The 0.1.0 version of CyborgFlow has been [released and is available for
download](https://github.com/SphereEx/CyborgFlow/releases).
+
+## Apache ShardingSphere Open Source Project Links:
+[ShardingSphere Github](https://github.com/apache/shardingsphere)
+
+[ShardingSphere Twitter](https://twitter.com/ShardingSphere)
+
+[ShardingSphere Slack
Channel](https://apacheshardingsphere.slack.com/join/shared_invite/zt-sbdde7ie-SjDqo9~I4rYcR18bq0SYTg)
+
+[Contributor Guide](https://shardingsphere.apache.org/community/cn/contribute/)
+
+## Author
+
+HouYang
+
+> SphereEx Middleware Developer, Apache ShardingSphere Contributor.
+>
+> Currently he focuses on the design and development of ShadowDB and full-link
stress testing.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_1.png
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_1.png
new file mode 100644
index 0000000..d4b4e52
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_1.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_10.png
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_10.png
new file mode 100644
index 0000000..66ed351
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_10.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_2.png
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_2.png
new file mode 100644
index 0000000..1f385c0
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_2.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_3.jpeg
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_3.jpeg
new file mode 100644
index 0000000..18ded1f
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_3.jpeg differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_4.png
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_4.png
new file mode 100644
index 0000000..250d89a
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_4.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_5.jpg
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_5.jpg
new file mode 100644
index 0000000..dc60d9d
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_5.jpg differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_6.png
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_6.png
new file mode 100644
index 0000000..3419390
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_6.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_7.png
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_7.png
new file mode 100644
index 0000000..9927e38
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_7.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_8.png
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_8.png
new file mode 100644
index 0000000..00a915d
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_8.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_9.png
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_9.png
new file mode 100644
index 0000000..f6e5bb0
Binary files /dev/null and
b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_9.png differ
diff --git a/docs/blog/static/img/Hou_Yang_Photo.png
b/docs/blog/static/img/Hou_Yang_Photo.png
new file mode 100644
index 0000000..281729b
Binary files /dev/null and b/docs/blog/static/img/Hou_Yang_Photo.png differ
