dosubot[bot] commented on issue #7508:
URL:
https://github.com/apache/incubator-devlake/issues/7508#issuecomment-2129170396
<!-- Answer -->
The `pull_requests` table might not be populating with all the repositories
added to your project due to the filtering conditions in the
`EnrichPullRequestIssues` function. Specifically, the `dal.Where` clause
filters the data based on `repo_id` and `connection_id`. If these IDs do not
match the repositories you expect to be included, the data will not be
processed.
Here is the relevant code snippet:
```go
cursor, err := db.Cursor(
dal.From(&models.GithubPullRequest{}),
dal.Where("repo_id = ? and connection_id = ?", repoId,
data.Options.ConnectionId),
)
```
Ensure that the `repoId` and `data.Options.ConnectionId` are correctly set
for all the repositories you have added to your project. If these values are
incorrect or not set for some repositories, the corresponding pull requests
will not be included in the `pull_requests` table.
Additionally, verify that the `PullRequest` struct in your codebase is
correctly defined and matches the expected schema. Here is the definition for
reference:
```go
type PullRequest struct {
DomainEntity
BaseRepoId string `gorm:"index"`
HeadRepoId string `gorm:"index"`
Status string `gorm:"type:varchar(100);comment:open/closed or
other"`
Title string
Description string
Url string `gorm:"type:varchar(255)"`
AuthorName string `gorm:"type:varchar(100)"`
AuthorId string `gorm:"type:varchar(100)"`
ParentPrId string `gorm:"index;type:varchar(100)"`
PullRequestKey int
CreatedDate time.Time
MergedDate *time.Time
ClosedDate *time.Time
Type string `gorm:"type:varchar(100)"`
Component string `gorm:"type:varchar(100)"`
MergeCommitSha string `gorm:"type:varchar(40)"`
HeadRef string `gorm:"type:varchar(255)"`
BaseRef string `gorm:"type:varchar(255)"`
BaseCommitSha string `gorm:"type:varchar(40)"`
HeadCommitSha string `gorm:"type:varchar(40)"`
}
func (PullRequest) TableName() string {
return "pull_requests"
}
```
Make sure that the table name and field types are correctly set to avoid any
schema mismatches.
Additionally, ensure that the `EnrichPullRequestIssues` function in your
code is correctly extracting and processing the pull request data. Here is the
relevant part of the `EnrichPullRequestIssues` function:
```go
func EnrichPullRequestIssues(taskCtx plugin.SubTaskContext) (err
errors.Error) {
db := taskCtx.GetDal()
data := taskCtx.GetData().(*GithubTaskData)
repoId := data.Options.GithubId
var prBodyCloseRegex *regexp.Regexp
prBodyClosePattern := data.Options.ScopeConfig.PrBodyClosePattern
prBodyClosePattern = strings.Replace(prBodyClosePattern, "%s",
data.Options.Name, 1)
if len(prBodyClosePattern) > 0 {
prBodyCloseRegex, err =
errors.Convert01(regexp.Compile(prBodyClosePattern))
if err != nil {
return errors.Default.Wrap(err, "regexp Compile
prBodyClosePattern failed")
}
}
charPattern := regexp.MustCompile(`[\/a-zA-Z\s,]+`)
cursor, err := db.Cursor(dal.From(&models.GithubPullRequest{}),
dal.Where("repo_id = ? and connection_id = ?", repoId,
data.Options.ConnectionId))
if err != nil {
return err
}
defer cursor.Close()
converter, err := api.NewDataConverter(api.DataConverterArgs{
InputRowType: reflect.TypeOf(models.GithubPullRequest{}),
Input: cursor,
RawDataSubTaskArgs: api.RawDataSubTaskArgs{
Ctx: taskCtx,
Params: GithubApiParams{
ConnectionId: data.Options.ConnectionId,
Name: data.Options.Name,
},
Table: RAW_PULL_REQUEST_TABLE,
},
Convert: func(inputRow interface{}) ([]interface{}, errors.Error) {
githubPullRequst := inputRow.(*models.GithubPullRequest)
results := make([]interface{}, 0, 1)
issueNumberListStr := ""
if prBodyCloseRegex != nil {
issueNumberListStr =
prBodyCloseRegex.FindString(githubPullRequst.Body)
}
if issueNumberListStr == "" {
return nil, nil
}
issueNumberListStr =
charPattern.ReplaceAllString(issueNumberListStr, "#")
issueNumberList := strings.Split(issueNumberListStr, "#")
for _, issueNumberStr := range issueNumberList {
issue := &models.GithubIssue{}
issueNumberStr = strings.TrimSpace(issueNumberStr)
issueNumber, numFormatErr := strconv.Atoi(issueNumberStr)
if numFormatErr != nil {
continue
}
err = db.All(
issue,
dal.Where("number = ? and repo_id = ? and connection_id
= ?",
issueNumber, repoId, data.Options.ConnectionId),
dal.Limit(1),
)
if err != nil {
return nil, err
}
if issue.Number == 0 {
continue
}
githubPullRequstIssue := &models.GithubPrIssue{
ConnectionId: data.Options.ConnectionId,
PullRequestId: githubPullRequst.GithubId,
IssueId: issue.GithubId,
PullRequestNumber: githubPullRequst.Number,
IssueNumber: issue.Number,
}
results = append(results, githubPullRequstIssue)
}
return results, nil
},
})
if err != nil {
return err
}
return converter.Execute()
}
```
Ensure that the `EnrichPullRequestIssues` function is correctly configured
and that the `ConnectionId` and `repoId` are properly set for all repositories.
This function is responsible for extracting and processing the pull request
data, and any misconfiguration here could result in missing data in the
`pull_requests` table
<sup>[[1]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/tasks/pr_issue_convertor.go)</sup><sup>[[2]](https://github.com/apache/incubator-devlake/blob/main/backend/core/models/migrationscripts/archived/pull_request.go)</sup><sup>[[3]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github_graphql/tasks/pr_extractor.go)</sup><sup>[[4]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/tasks/pr_issue_enricher.go)</sup>.
<!-- Dosu Comment Footer -->
<details>
<summary>
#### Details
</summary>
**About Dosu**
This response is meant to be useful and save you time. It is **not meant to
be a precise solution**, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is
irrelevant.
</details>
*To continue the conversation, mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
