likyh opened a new pull request, #3705:
URL: https://github.com/apache/incubator-devlake/pull/3705
# Summary
<!--
Thanks for submitting a pull request!
We appreciate you spending the time to work on these changes.
Please fill out as many sections below as possible.
-->
CheckSuite.CheckRuns have some columns which are similar to WorkflowJobs.
So collect `CheckRuns` on each CheckSuite to replace WorkflowJobCollector.
Now query 30 CheckSuite in one graphql request. This new collector costs
about 40min to collect Devlake's runs & jobs on my computer, and its speed is
about 1000/min (CheckSuite) or 2 requests per second.
### Does this close any open issues?
Closes #3704
### Screenshots
The result in `_tool_github_jobs`:
<table border="1" style="border-collapse:collapse">
<tr><th>created_at</th><th>updated_at</th><th>_raw_data_params</th><th>_raw_data_table</th><th>_raw_data_id</th><th>_raw_data_remark</th><th>connection_id</th><th>repo_id</th><th>id</th><th>run_id</th><th>run_url</th><th>node_id</th><th>head_sha</th><th>url</th><th>html_url</th><th>status</th><th>conclusion</th><th>started_at</th><th>completed_at</th><th>name</th><th>steps</th><th>check_run_url</th><th>labels</th><th>runner_id</th><th>runner_name</th><th>runner_group_id</th><th>type</th></tr>
<tr><td>2022-11-09 14:08:19.001904 +00:00</td><td>2022-11-09 14:08:19.001904
+00:00</td><td>{"ConnectionId":1,"Owner":"apache","Repo":"incubator-devlake"}</td><td>_raw_github_graphql_check_runs</td><td>280</td><td></td><td>1</td><td>384111310</td><td>9386708808</td><td>0</td><td></td><td>CR_kwDOFuUSzs8AAAACL33PSA</td><td></td><td></td><td>https://github.com/apache/incubator-devlake/actions/runs/3428125760/jobs/5711986448</td><td>COMPLETED</td><td>SUCCESS</td><td>2022-11-09
12:25:46.000000 +00:00</td><td>2022-11-09 12:28:36.000000 +00:00</td><td>test
(ubuntu-latest)</td><td>[{"Name": "Set up job",
"Number": 1, "Status": "COMPLETED",
"StartedAt": "2022-11-09T12:25:46Z",
"Conclusion": "SUCCESS", "ExternalId":
"498244a7-bb2d-4a89-b0e6-a02636e00f67", "CompletedAt":
"2022-11-09T12:25:47Z", "SecondsToCompl
etion": 1}, {"Name": "Initialize containers",
"Number": 2, "Status": "COMPLETED",
"StartedAt": "2022-11-09T12:25:47Z",
"Conclusion": "SUCCESS", "ExternalId":
"82ed782a-533a-443d-b9b3-08973755133f", "CompletedAt":
"2022-11-09T12:26:03Z", "SecondsToCompletion": 16},
{"Name": "Checkout code", "Number": 3,
"Status": "COMPLETED", "StartedAt":
"2022-11-09T12:26:03Z", "Conclusion": "SUCCESS",
"ExternalId": "dae6463b-ccc6-5762-2432-19367673d22a",
"CompletedAt": "2022-11-09T12:26:08Z",
"SecondsToCompletion": 5}, {"Name": "Setup Golang
env", "Number": 4, "Status": "COMPLETED",
"StartedAt": "2022-11-09T12:26:08Z",
"Conclusion": "S
UCCESS", "ExternalId":
"36275aed-210b-514e-91f2-23c7b347dfa3", "CompletedAt":
"2022-11-09T12:26:09Z", "SecondsToCompletion": 1},
{"Name": "Cache unit-test", "Number": 5,
"Status": "COMPLETED", "StartedAt":
"2022-11-09T12:26:09Z", "Conclusion": "SUCCESS",
"ExternalId": "f6f7da5e-8567-5f82-1a7e-cf9ca6c0ba8c",
"CompletedAt": "2022-11-09T12:26:17Z",
"SecondsToCompletion": 8}, {"Name": "Unit tests",
"Number": 6, "Status": "COMPLETED",
"StartedAt": "2022-11-09T12:26:17Z",
"Conclusion": "SUCCESS", "ExternalId":
"ff0bb0fc-e09e-55fc-4b83-ae5ebb72a959", "CompletedAt":
"2022-11-09T12:28:32Z", "SecondsToCompletion": 135},
{"Name": "Post Cache unit-tes
t", "Number": 9, "Status": "COMPLETED",
"StartedAt": "2022-11-09T12:28:32Z",
"Conclusion": "SUCCESS", "ExternalId":
"9785e0d4-de7d-46ff-9d2a-148c27bdf52c", "CompletedAt":
"2022-11-09T12:28:33Z", "SecondsToCompletion": 1},
{"Name": "Post Setup Golang env", "Number": 10,
"Status": "COMPLETED", "StartedAt":
"2022-11-09T12:28:33Z", "Conclusion": "SUCCESS",
"ExternalId": "73c14928-e3cd-4574-9411-f324bf8d9424",
"CompletedAt": "2022-11-09T12:28:33Z",
"SecondsToCompletion": 0}, {"Name": "Post Checkout
code", "Number": 11, "Status": "COMPLETED",
"StartedAt": "2022-11-09T12:28:33Z",
"Conclusion": "SUCCESS", "ExternalId":
"ce92c31
0-bd83-41cf-8bb8-2de5e9647bdc", "CompletedAt":
"2022-11-09T12:28:33Z", "SecondsToCompletion": 0},
{"Name": "Stop containers", "Number": 12,
"Status": "COMPLETED", "StartedAt":
"2022-11-09T12:28:34Z", "Conclusion": "SUCCESS",
"ExternalId": "990c54f2-653a-487c-9c26-1025a0b41a86",
"CompletedAt": "2022-11-09T12:28:35Z",
"SecondsToCompletion": 1}, {"Name": "Complete
job", "Number": 13, "Status": "COMPLETED",
"StartedAt": "2022-11-09T12:28:35Z",
"Conclusion": "SUCCESS", "ExternalId":
"bfea7717-cc32-4594-ae01-d1570b771e47", "CompletedAt":
"2022-11-09T12:28:35Z", "SecondsToCompletion":
0}]</td><td></td><td>NULL</td><td>0</td><td></td><td>0</td><td></td></tr>
<tr><td>2022-11-09 14:08:19.001904 +00:00</td><td>2022-11-09 14:08:19.001904
+00:00</td><td>{"ConnectionId":1,"Owner":"apache","Repo":"incubator-devlake"}</td><td>_raw_github_graphql_check_runs</td><td>280</td><td></td><td>1</td><td>384111310</td><td>9385401532</td><td>0</td><td></td><td>CR_kwDOFuUSzs8AAAACL2ncvA</td><td></td><td></td><td>https://github.com/apache/incubator-devlake/actions/runs/3427686139/jobs/5710992069</td><td>COMPLETED</td><td>SUCCESS</td><td>2022-11-09
11:18:09.000000 +00:00</td><td>2022-11-09 11:22:58.000000
+00:00</td><td>e2e-mysql</td><td>[{"Name": "Set up job",
"Number": 1, "Status": "COMPLETED",
"StartedAt": "2022-11-09T11:18:08Z",
"Conclusion": "SUCCESS", "ExternalId":
"13d3eb72-233a-46e8-b033-6dc9db7520c1", "CompletedAt":
"2022-11-09T11:18:10Z", "SecondsToCompletion"
: 2}, {"Name": "Initialize containers",
"Number": 2, "Status": "COMPLETED",
"StartedAt": "2022-11-09T11:18:10Z",
"Conclusion": "SUCCESS", "ExternalId":
"eac729a0-9092-4140-8b19-9398a560b028", "CompletedAt":
"2022-11-09T11:18:44Z", "SecondsToCompletion": 34},
{"Name": "Run actions/checkout@v3", "Number": 3,
"Status": "COMPLETED", "StartedAt":
"2022-11-09T11:18:44Z", "Conclusion": "SUCCESS",
"ExternalId": "70b9334f-e9db-5c1a-f7db-4250a47e57f2",
"CompletedAt": "2022-11-09T11:18:48Z",
"SecondsToCompletion": 4}, {"Name": "Cache
test-e2e", "Number": 4, "Status":
"COMPLETED", "StartedAt": "2022-11-09T11:18:48Z",
"Conclusion": "SUCC
ESS", "ExternalId":
"36b05d30-1014-5f7d-aad2-c4aee0d7ec30", "CompletedAt":
"2022-11-09T11:19:02Z", "SecondsToCompletion": 14},
{"Name": "Test", "Number": 5, "Status":
"COMPLETED", "StartedAt": "2022-11-09T11:19:02Z",
"Conclusion": "SUCCESS", "ExternalId":
"18a23b63-b968-5ac0-3668-34310a7a8e14", "CompletedAt":
"2022-11-09T11:22:53Z", "SecondsToCompletion": 231},
{"Name": "Post Cache test-e2e", "Number": 8,
"Status": "COMPLETED", "StartedAt":
"2022-11-09T11:22:53Z", "Conclusion": "SUCCESS",
"ExternalId": "b84ced8a-4679-4c36-95be-38b77b097724",
"CompletedAt": "2022-11-09T11:22:54Z",
"SecondsToCompletion": 1}, {"Name": "Post Run
actions/checko
ut@v3", "Number": 9, "Status": "COMPLETED",
"StartedAt": "2022-11-09T11:22:54Z",
"Conclusion": "SUCCESS", "ExternalId":
"0741c6c7-c5fd-47c6-8963-beb995598b71", "CompletedAt":
"2022-11-09T11:22:54Z", "SecondsToCompletion": 0},
{"Name": "Stop containers", "Number": 10,
"Status": "COMPLETED", "StartedAt":
"2022-11-09T11:22:55Z", "Conclusion": "SUCCESS",
"ExternalId": "e08816f4-38a4-4af2-934d-3b55f3290812",
"CompletedAt": "2022-11-09T11:22:57Z",
"SecondsToCompletion": 2}, {"Name": "Complete
job", "Number": 11, "Status": "COMPLETED",
"StartedAt": "2022-11-09T11:22:57Z",
"Conclusion": "SUCCESS", "ExternalId":
"f549758f-8e87-4
de5-b3bd-0e1ac661bb96", "CompletedAt":
"2022-11-09T11:22:57Z", "SecondsToCompletion":
0}]</td><td></td><td>NULL</td><td>0</td><td></td><td>0</td><td></td></tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
