This is an automated email from the ASF dual-hosted git repository. zky pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
commit 6fecb70b7bc1cb5125e2536d92c637e6c1ffe0ac Author: linyh <[email protected]> AuthorDate: Mon Jun 27 14:13:35 2022 +0800 update for new generator --- ...2e-test-writing-guide-zn.md => e2e-test-writing-guide.md} | 11 +++++++---- .../current/09-DeveloperDoc/e2e-test-writing-guide.md | 12 ++++++++---- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/docs/09-DeveloperDoc/e2e-test-writing-guide-zn.md b/docs/09-DeveloperDoc/e2e-test-writing-guide.md similarity index 91% rename from docs/09-DeveloperDoc/e2e-test-writing-guide-zn.md rename to docs/09-DeveloperDoc/e2e-test-writing-guide.md index ec916e54..2f45648a 100644 --- a/docs/09-DeveloperDoc/e2e-test-writing-guide-zn.md +++ b/docs/09-DeveloperDoc/e2e-test-writing-guide.md @@ -2,7 +2,7 @@ ## Why write E2E tests -E2E testing, as a part of automated testing, generally refers to white-box integration testing or unit testing that allows the use of some external services such as databases. The purpose of writing E2E tests is to shield some internal implementation logic and see whether the same external input can output the same result in terms of data aspects. In addition, compared to the black-box integration tests, it can avoid some chance problems caused by network and other factors. More informat [...] +E2E testing, as a part of automated testing, generally refers to black-box testing at the file and module level or unit testing that allows the use of some external services such as databases. The purpose of writing E2E tests is to shield some internal implementation logic and see whether the same external input can output the same result in terms of data aspects. In addition, compared to the black-box integration tests, it can avoid some chance problems caused by network and other facto [...] In DevLake, E2E testing consists of interface testing and input/output result validation for the plugin Extract/Convert subtask. This article only describes the process of writing the latter. ## Preparing data @@ -40,15 +40,18 @@ Next, we need to export the data to .csv format. This step is a variety of optio ### DevLake Code Generator Export -This program is not yet completed +Run `go run generator/main.go create-e2e-raw` directly and follow the guidelines to complete the export. This solution is the simplest, but has some limitations, such as the exported fields are fixed. If you need more customization options, you can refer to next solutions. + + ### GoLand Database export  -This solution is the easiest to use and will not cause any problems using Postgres or MySQL. +This solution is very easy to use and will not cause any problems using Postgres or MySQL.  The success criteria for csv export is that the go program can read it without errors, so several points are worth noticing. + 1. the values in the csv file should be wrapped in double quotes to avoid special symbols such as commas in the values that break the csv format 2. double quotes in csv files are escaped. generally `""` represents a double quote 3. pay attention to whether the column `data` is the actual value, not the value after base64 or hex @@ -131,7 +134,7 @@ func TestMeetingDataFlow(t *testing.T) { } ``` -The added code includes a call to `dataflowTester.FlushTabler` to clear the meeting table and a call to `dataflowTester.Subtask` to simulate the running of the subtask `ExtractMeetingTopUserItemMeta`. +The added code includes a call to `dataflowTester.FlushTabler` to clear the table `_tool_feishu_meeting_top_user_items` and a call to `dataflowTester.Subtask` to simulate the running of the subtask `ExtractMeetingTopUserItemMeta`. Now run it and see if the subtask `ExtractMeetingTopUserItemMeta` completes without errors. The data results of the `extract` run generally come from the raw table, so the plugin subtask will run correctly if it is written without errors. You can observe if the data is successfully parsed in the db table in the tool layer. In this case the `_tool_feishu_meeting_top_user_items` table has the correct data. diff --git a/docs/09-DeveloperDoc/e2e-test-writing-guide-cn.md b/i18n/zh/docusaurus-plugin-content-docs/current/09-DeveloperDoc/e2e-test-writing-guide.md similarity index 90% rename from docs/09-DeveloperDoc/e2e-test-writing-guide-cn.md rename to i18n/zh/docusaurus-plugin-content-docs/current/09-DeveloperDoc/e2e-test-writing-guide.md index cf5a2c92..b664a6ca 100644 --- a/docs/09-DeveloperDoc/e2e-test-writing-guide-cn.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/09-DeveloperDoc/e2e-test-writing-guide.md @@ -2,7 +2,7 @@ ## 为什么要写 E2E 测试 -E2E 测试，作为自动化测试的一环，一般是指白盒的集成测试，或者允许使用一些数据库等外部服务的单元测试。书写E2E测试的目的是屏蔽一些内部实现逻辑，仅从数据正确性的角度来看同样的外部输入，是否可以得到同样的输出。另外，相较于黑盒的集成测试来说，可以避免一些网络等因素带来的偶然问题。更多关于插件的介绍，可以在这里获取更多信息： 为什么要编写 E2E 测试（未完成） +E2E 测试，作为自动化测试的一环，一般是指文件与模块级别的黑盒测试，或者允许使用一些数据库等外部服务的单元测试。书写E2E测试的目的是屏蔽一些内部实现逻辑，仅从数据正确性的角度来看同样的外部输入，是否可以得到同样的输出。另外，相较于黑盒的集成测试来说，可以避免一些网络等因素带来的偶然问题。更多关于插件的介绍，可以在这里获取更多信息： 为什么要编写 E2E 测试（未完成） 在 DevLake 中，E2E 测试包含接口测试和插件 Extract/Convert 子任务的输入输出结果验证，本篇仅介绍后者的编写流程。 ## 准备数据 @@ -13,6 +13,7 @@ E2E 测试，作为自动化测试的一环，一般是指白盒的集成测试 编写测试的第一步，就是运行一下对应插件的 Collect 任务，完成数据的收集，也就是让数据库的`_raw_feishu_`开头的表中，保存有对应的数据。 以下是采用 DirectRun (cmd) 运行方式的运行日志和数据库结果。 + ``` $ go run plugins/feishu/main.go --numOfDaysToCollect 2 --connectionId 1 （注意：随着版本的升级，命令可能产生变化） [2022-06-22 23:03:29] INFO failed to create dir logs: mkdir logs: file exists @@ -40,15 +41,18 @@ press `c` to send cancel signal ### DevLake Code Generator 导出 -此方案尚未完工 +直接运行`go run generator/main.go create-e2e-raw`，根据指引来完成导出。此方案最简单，但也有一定的局限性，比如导出的字段是固定的，如果需要更多的自定义选项，可以参考接下来的方案。 + + ### GoLand Database 导出  -这种方案是最简单的，无论使用Postgres或者MySQL，都不会出现什么问题。 +这种方案很简单，无论使用Postgres或者MySQL，都不会出现什么问题。  csv导出的成功标准就是go程序可以无误的读取，因此有以下几点值得注意： + 1. csv文件中的值，可以用双引号包裹，避免值中的逗号等特殊符号破坏了csv格式 2. csv文件中双引号转义，一般是`""`代表一个双引号 3. 注意观察data是否是真实值，而不是base64后的值 @@ -133,7 +137,7 @@ func TestMeetingDataFlow(t *testing.T) { } ``` -新增的代码包括调用`dataflowTester.FlushTabler`刷新会议表，调用`dataflowTester.Subtask`模拟子任务`ExtractMeetingTopUserItemMeta`的运行。 +新增的代码包括调用`dataflowTester.FlushTabler`清空`FeishuMeetingTopUserItem`对应的表，调用`dataflowTester.Subtask`模拟子任务`ExtractMeetingTopUserItemMeta`的运行。 现在在运行试试吧，看看子任务`ExtractMeetingTopUserItemMeta`是否能没有错误的完成运行。`extract`运行的数据结果一般来自raw表，因此，插件子任务编写如果没有差错的话，会正确运行，并且可以在 toolLayer 层的数据表中观察到数据成功解析，在本案例中，即`_tool_feishu_meeting_top_user_items`表中有正确的数据。
