dosubot[bot] commented on issue #7508: URL: https://github.com/apache/incubator-devlake/issues/7508#issuecomment-2129170396
<!-- Answer --> The `pull_requests` table might not be populating with all the repositories added to your project due to the filtering conditions in the `EnrichPullRequestIssues` function. Specifically, the `dal.Where` clause filters the data based on `repo_id` and `connection_id`. If these IDs do not match the repositories you expect to be included, the data will not be processed. Here is the relevant code snippet: ```go cursor, err := db.Cursor( dal.From(&models.GithubPullRequest{}), dal.Where("repo_id = ? and connection_id = ?", repoId, data.Options.ConnectionId), ) ``` Ensure that the `repoId` and `data.Options.ConnectionId` are correctly set for all the repositories you have added to your project. If these values are incorrect or not set for some repositories, the corresponding pull requests will not be included in the `pull_requests` table. Additionally, verify that the `PullRequest` struct in your codebase is correctly defined and matches the expected schema. Here is the definition for reference: ```go type PullRequest struct { DomainEntity BaseRepoId string `gorm:"index"` HeadRepoId string `gorm:"index"` Status string `gorm:"type:varchar(100);comment:open/closed or other"` Title string Description string Url string `gorm:"type:varchar(255)"` AuthorName string `gorm:"type:varchar(100)"` AuthorId string `gorm:"type:varchar(100)"` ParentPrId string `gorm:"index;type:varchar(100)"` PullRequestKey int CreatedDate time.Time MergedDate *time.Time ClosedDate *time.Time Type string `gorm:"type:varchar(100)"` Component string `gorm:"type:varchar(100)"` MergeCommitSha string `gorm:"type:varchar(40)"` HeadRef string `gorm:"type:varchar(255)"` BaseRef string `gorm:"type:varchar(255)"` BaseCommitSha string `gorm:"type:varchar(40)"` HeadCommitSha string `gorm:"type:varchar(40)"` } func (PullRequest) TableName() string { return "pull_requests" } ``` Make sure that the table name and field types are correctly set to avoid any schema mismatches. Additionally, ensure that the `EnrichPullRequestIssues` function in your code is correctly extracting and processing the pull request data. Here is the relevant part of the `EnrichPullRequestIssues` function: ```go func EnrichPullRequestIssues(taskCtx plugin.SubTaskContext) (err errors.Error) { db := taskCtx.GetDal() data := taskCtx.GetData().(*GithubTaskData) repoId := data.Options.GithubId var prBodyCloseRegex *regexp.Regexp prBodyClosePattern := data.Options.ScopeConfig.PrBodyClosePattern prBodyClosePattern = strings.Replace(prBodyClosePattern, "%s", data.Options.Name, 1) if len(prBodyClosePattern) > 0 { prBodyCloseRegex, err = errors.Convert01(regexp.Compile(prBodyClosePattern)) if err != nil { return errors.Default.Wrap(err, "regexp Compile prBodyClosePattern failed") } } charPattern := regexp.MustCompile(`[\/a-zA-Z\s,]+`) cursor, err := db.Cursor(dal.From(&models.GithubPullRequest{}), dal.Where("repo_id = ? and connection_id = ?", repoId, data.Options.ConnectionId)) if err != nil { return err } defer cursor.Close() converter, err := api.NewDataConverter(api.DataConverterArgs{ InputRowType: reflect.TypeOf(models.GithubPullRequest{}), Input: cursor, RawDataSubTaskArgs: api.RawDataSubTaskArgs{ Ctx: taskCtx, Params: GithubApiParams{ ConnectionId: data.Options.ConnectionId, Name: data.Options.Name, }, Table: RAW_PULL_REQUEST_TABLE, }, Convert: func(inputRow interface{}) ([]interface{}, errors.Error) { githubPullRequst := inputRow.(*models.GithubPullRequest) results := make([]interface{}, 0, 1) issueNumberListStr := "" if prBodyCloseRegex != nil { issueNumberListStr = prBodyCloseRegex.FindString(githubPullRequst.Body) } if issueNumberListStr == "" { return nil, nil } issueNumberListStr = charPattern.ReplaceAllString(issueNumberListStr, "#") issueNumberList := strings.Split(issueNumberListStr, "#") for _, issueNumberStr := range issueNumberList { issue := &models.GithubIssue{} issueNumberStr = strings.TrimSpace(issueNumberStr) issueNumber, numFormatErr := strconv.Atoi(issueNumberStr) if numFormatErr != nil { continue } err = db.All( issue, dal.Where("number = ? and repo_id = ? and connection_id = ?", issueNumber, repoId, data.Options.ConnectionId), dal.Limit(1), ) if err != nil { return nil, err } if issue.Number == 0 { continue } githubPullRequstIssue := &models.GithubPrIssue{ ConnectionId: data.Options.ConnectionId, PullRequestId: githubPullRequst.GithubId, IssueId: issue.GithubId, PullRequestNumber: githubPullRequst.Number, IssueNumber: issue.Number, } results = append(results, githubPullRequstIssue) } return results, nil }, }) if err != nil { return err } return converter.Execute() } ``` Ensure that the `EnrichPullRequestIssues` function is correctly configured and that the `ConnectionId` and `repoId` are properly set for all repositories. This function is responsible for extracting and processing the pull request data, and any misconfiguration here could result in missing data in the `pull_requests` table <sup>[[1]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/tasks/pr_issue_convertor.go)</sup><sup>[[2]](https://github.com/apache/incubator-devlake/blob/main/backend/core/models/migrationscripts/archived/pull_request.go)</sup><sup>[[3]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github_graphql/tasks/pr_extractor.go)</sup><sup>[[4]](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github/tasks/pr_issue_enricher.go)</sup>. <!-- Dosu Comment Footer --> <details> <summary> #### Details </summary> **About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. </details> *To continue the conversation, mention @dosu.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org