This is an automated email from the ASF dual-hosted git repository.
likyh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new 287796f74 docs: update plugin implementation and refdiff (#325)
287796f74 is described below
commit 287796f74799010484d56c293d33f7762ff3e842
Author: Louis.z <[email protected]>
AuthorDate: Wed Nov 16 16:51:52 2022 +0800
docs: update plugin implementation and refdiff (#325)
Co-authored-by: Startrekzky <[email protected]>
---
docs/DeveloperManuals/PluginImplementation.md | 22 +++++++++++-----------
docs/Plugins/refdiff.md | 21 +++++++++++----------
2 files changed, 22 insertions(+), 21 deletions(-)
diff --git a/docs/DeveloperManuals/PluginImplementation.md
b/docs/DeveloperManuals/PluginImplementation.md
index 34f85ef0e..e1cf04cec 100644
--- a/docs/DeveloperManuals/PluginImplementation.md
+++ b/docs/DeveloperManuals/PluginImplementation.md
@@ -36,8 +36,8 @@ A plugin mainly consists of a collection of subtasks that can
be executed by Dev
3.
[PluginTask](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_task.go)
enables a plugin to prepare data prior to subtask execution
4.
[PluginApi](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_api.go)
lets a plugin exposes some self-defined APIs
5.
[PluginMigration](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_migration.go)
is where a plugin manages its database migrations
-6.
[PluginModel](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_model.go)
allows other plugins to get the model information of all database tables of
the current plugin through the GetTablesInfo() method.If you need to access
Domain Layer Models,please visit
[DomainLayerSchema](https://devlake.apache.org/docs/DataModels/DevLakeDomainLayerSchema/)
-7.
[PluginBlueprint](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_blueprint.go)
is the foundation for Blueprint and Plugin to collaborate and generate a
reasonable Pipeline Plan based on User Settings. For example, a user may
declare that he wants to collect data from a Github Repo, which implies that
not only the issues/prs, but also the git-meta-data including commits history,
branches, tags, etc. needs to be collected. In order to do it and do it faster,
l [...]
+6.
[PluginModel](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_model.go)
allows other plugins to get the model information of all database tables of
the current plugin through the GetTablesInfo() method. If you need to access
Domain Layer Models, please visit
[DomainLayerSchema](https://devlake.apache.org/docs/DataModels/DevLakeDomainLayerSchema/)
+7.
[PluginBlueprint](https://github.com/apache/incubator-devlake/blob/main/plugins/core/plugin_blueprint.go)
is the foundation for Blueprint and Plugin to collaborate and generate a
reasonable Pipeline Plan based on User Settings. For example, a user may
declare that he/she wants to collect data from a GitHub Repo, which implies
that not only the issues and PRs, but also the git-meta-data including commits
history, branches, tags, etc. need to be collected. To do it and do it faster,
lev [...]
The diagram below shows the control flow of executing a plugin:
@@ -76,7 +76,7 @@ flowchart TD;
```
There's a lot of information in the diagram, but we don't expect you to digest
it right away. You can simply use it as a reference when you go through the
example below.
-## A step-by-step guide towards your first conventional plugin
+## A step-by-step guide toward your first conventional plugin
In this section, we will describe how to create a data collection plugin from
scratch. The data to be collected is the information about all Committers and
Contributors of the Apache project, in order to check whether they have signed
the CLA. We are going to
@@ -101,19 +101,19 @@ We will focus on demonstrating how to request and cache
information about all Co
> - task data
> [template](https://github.com/apache/incubator-devlake/blob/main/generator/template/plugin/tasks/task_data.go-template)
> - api client
> [template](https://github.com/apache/incubator-devlake/blob/main/generator/template/plugin/tasks/task_data_with_api_client.go-template)
-Don't worry if you cannot figure out what these concepts mean immediately.
We'll explain them one by one later.
+Don't worry if you cannot figure out what these concepts mean immediately.
We'll explain them one by one later.
-DevLake provides a generator to create a plugin conveniently. Let's scaffold
our new plugin by running `go run generator/main.go create-plugin icla`, which
would ask for `with_api_client` and `Endpoint`.
+Apache DevLake provides a generator to create a plugin conveniently. Let's
scaffold our new plugin by running `go run generator/main.go create-plugin
icla`, which would ask for `with_api_client` and `Endpoint`.
* `with_api_client` is used for choosing if we need to request HTTP APIs by
api_client.
* `Endpoint` use in which site we will request, in our case, it should be
`https://people.apache.org/`.
-Now we have three files in our plugin. `api_client.go` and `task_data.go` are
in subfolder `tasks/`.
+Now we have three files in our plugin. `api_client.go` and `task_data.go` are
in the subfolder `tasks/`.
-Have a try to run this plugin by function `main` in `plugin_main.go`. When you
see result like this:
+Have a try to run this plugin by function `main` in `plugin_main.go`. When you
see results like this:
```
$go run plugins/icla/plugin_main.go
[2022-06-02 18:07:30] INFO failed to create dir logs: mkdir logs: file exists
@@ -123,10 +123,10 @@ invalid ICLA_TOKEN, but ignore this error now
[2022-06-02 18:07:30] INFO [icla] scheduler for api
https://people.apache.org/ worker: 25, request: 18000, duration: 1h0m0s
[2022-06-02 18:07:30] INFO [icla] total step: 0
```
-How exciting. It works! The plugin defined and initiated in `plugin_main.go`
use some options in `task_data.go`. They are made up as the most
straightforward plugin in Apache DevLake, and `api_client.go` will be used in
the next step to request HTTP APIs.
+It works! Plugin 'icla' is defined and initiated. This plugin ONLY contains
`plugin_main.go` and `task_data.go`, which is the simplest form of a plugin in
Apache DevLake. In the next step, we'll show you how to request HTTP APIs by
`api_client.go`.
### Step 2: Create a sub-task for data collection
-Before we start, it is helpful to know how collection task is executed:
+Before we start, it is helpful to know how a collection task is executed:
1. First, Apache DevLake would call `plugin_main.PrepareTaskData()` to prepare
needed data before any sub-tasks. We need to create an API client here.
2. Then Apache DevLake will call the sub-tasks returned by
`plugin_main.SubTaskMetas()`. Sub-task is an independent task to do some job,
like requesting API, processing data, etc.
@@ -145,7 +145,7 @@ Let's run `go run generator/main.go create-collector icla
committer` and confirm
> - Collector will collect data from HTTP or other data sources, and save the
> data into the raw layer.
> - Inside the func `SubTaskEntryPoint` of `Collector`, we use
> `helper.NewApiCollector` to create an object of
> [ApiCollector](https://github.com/apache/incubator-devlake/blob/main/generator/template/plugin/tasks/api_collector.go-template),
> then call `execute()` to do the job.
-Now you can notice `data.ApiClient` is initiated in
`plugin_main.go/PrepareTaskData.ApiClient`. `PrepareTaskData` create a new
`ApiClient`, which is a tool Apache DevLake suggests to request data from HTTP
Apis. This tool support some valuable features for HttpApi, like rateLimit,
proxy and retry. Of course, if you like, you may use the lib `http` instead,
but it will be more tedious.
+Now you can notice `data.ApiClient` is initiated in
`plugin_main.go/PrepareTaskData.ApiClient`. `PrepareTaskData` creates a new
`ApiClient`, which is a tool Apache DevLake suggests to request data from HTTP
Apis. This tool support some valuable features for HttpApi, like rateLimit,
proxy and retry. Of course, if you like, you may use the lib `http` instead,
but it will be more tedious.
Let's move forward to use it.
@@ -206,7 +206,7 @@ receive data: 272956 /* <- the number means 272956 models
received */
We have already collected data from HTTP API and saved them into the DB table
`_raw_XXXX`. In this step, we will extract the names of committers from the raw
data. As you may infer from the name, raw tables are temporary and not easy to
use directly.
-Now Apache DevLake suggests to save data by
[gorm](https://gorm.io/docs/index.html), so we will create a model by gorm and
add it into `plugin_main.go/AutoMigrate()`.
+Now Apache DevLake suggests saving data by
[gorm](https://gorm.io/docs/index.html), so we will create a model by gorm and
add it into `plugin_main.go/AutoMigrate()`.
plugins/icla/models/committer.go
```go
diff --git a/docs/Plugins/refdiff.md b/docs/Plugins/refdiff.md
index 93a6cff8d..d97f1c1da 100644
--- a/docs/Plugins/refdiff.md
+++ b/docs/Plugins/refdiff.md
@@ -7,19 +7,19 @@ description: >
## Summary
-For development workload analysis, we often need to know how many commits have
been created between 2 releases. This plugin calculates which commits differ
between 2 Ref (branch/tag), and the result will be stored back into database
for further analysis.
+For development workload analysis, we often need to know how many commits have
been created between 2 releases. This plugin calculates which commits differ
between 2 Ref (branch/tag), and the result will be stored back into the
database for further analysis.
## Important Note
-You need to run gitextractor before the refdiff plugin. The gitextractor
plugin should create records in the `refs` table in your DB before this plugin
can be run.
+You need to run `gitextractor` before the `refdiff` plugin. The `gitextractor`
plugin should create records in the `refs` table in your database before this
plugin can be run.
## Configuration
-This is an enrichment plugin based on Domain Layer data, no configuration
needed
+This is an enrichment plugin based on the domain layer data, no configuration
is needed.
-## How to use
+## How to use refdiff
-In order to trigger the enrichment, you need to insert a new task into your
pipeline.
+To trigger the enrichment, you need to insert a new task into your pipeline.
1. Make sure `commits` and `refs` are collected into your database, `refs`
table should contain records like following:
```
@@ -35,7 +35,7 @@ In order to trigger the enrichment, you need to insert a new
task into your pipe
github:GithubRepo:384111310:refs/tags/v0.6.1 TAG
```
2. If you want to run calculatePrCherryPick, please configure
GITHUB_PR_TITLE_PATTERN in .env, you can check the example in .env.example(we
have a default value, please make sure your pattern is disclosed by single
quotes '')
-3. And then, trigger a pipeline like following, you can also define sub-tasks,
calculateRefDiff will calculate commits between two ref, and creatRefBugStats
will create a table to show bug list between two ref:
+3. And then, trigger a pipeline like the following format, you can also define
sub-tasks, calculateRefDiff will calculate commits between two refs, and
creatRefBugStats will create a table to show a bug list between two refs:
```shell
curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
{
@@ -89,7 +89,8 @@ curl -v -XPOST http://localhost:8080/pipelines --data @-
<<'JSON'
JSON
```
-4. Refdiff has a new subtask `calculateProjectDeploymentCommitsDiff`, which
can find related deployments according to the project, and then calculate the
commits diff between deployments. Such a function can be used to support the
calculation of dora indicators.
+## How to use refdiff in DORA
+RefDiff can be called by the [DORA
plugin](https://github.com/apache/incubator-devlake/tree/main/plugins/dora) to
support the calculation of [DORA
metrics](https://devlake.apache.org/docs/UserManuals/DORA). RefDiff has a
subtask called 'calculateProjectDeploymentCommitsDiff'. This subtask takes the
`project_name` from task options to calculate the commits diff between two
consecutive deployments in this project. That is to say, refdiff will generate
the relationship between `deployed com [...]
```shell
curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
@@ -113,7 +114,7 @@ curl -v -XPOST http://localhost:8080/pipelines --data @-
<<'JSON'
## Development
-This plugin depends on `libgit2`, you need to install version 1.3.0 in order
to run and debug this plugin on your local
+This plugin depends on `libgit2`, you need to install version 1.3.0 to run and
debug this plugin on your local
machine.
### Linux
@@ -139,7 +140,7 @@ export
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
```
-#### Troubleshooting (linux)
+#### Troubleshooting (Linux)
> Q: # pkg-config --cflags -- libgit2 Package libgit2 was not found in the
> pkg-config search path.
Perhaps you should add the directory containing `libgit2.pc` to the
PKG_CONFIG_PATH environment variable
@@ -159,7 +160,7 @@ No package 'libgit2' found pkg-config: exit status 1
### MacOS
-NOTE：Do **NOT** install libgit2 via `MacPorts` or `homebrew`, install from
source instead.
+NOTE: **Do NOT** install libgit2 via `MacPorts` or `homebrew`, install from
source instead.
```
brew install cmake
git clone https://github.com/libgit2/libgit2.git
