lrf-nitro opened a new pull request, #8858:
URL: https://github.com/apache/incubator-devlake/pull/8858

   <!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
   -->
   ### ⚠️ Pre Checklist
   
   > Please complete _ALL_ items in this checklist, and remove before submitting
   
   - [x] I have read through the [Contributing 
Documentation](https://devlake.apache.org/community/).
   - [x] I have added relevant tests.
   - [x] I have added relevant documentation.
   - [x] I will add labels to the PR, such as `pr-type/bug-fix`, 
`pr-type/feature-development`, etc.
   
   <!--
   Thanks for submitting a pull request!
   
   We appreciate you spending the time to work on these changes.
   Please fill out as many sections below as possible.
   -->
   
   ### Summary
   - This change set stabilizes GitHub incremental sync by moving high-volume 
extractors/converters to stateful incremental mode, hardening workflow-run 
collection windows, and fixing converter bootstrap/performance edge cases.
   - It also improves observability with per-subtask processed-record counts in 
completion logs.
   - Key fixes:
     - Stateful extractor + converter migration for GitHub plugin.
     - Time format change to RFC3339 since formatting for GitHub API collectors.
     - Workflow runs incremental bootstrap fallback + adaptive-window safety 
retained.
     - Subtask-state bootstrap from collector checkpoints to avoid broad 
converter first-pass on long-lived projects.
     - Convert Jobs incremental query optimized to avoid expensive run-table 
join scans on empty deltas.
   - Observed runtime outcomes from the logs:
     - Pre-fix data collection run time approx. 2 hours
     - With incremental handling the processing time was reduced to 2 minutes
   - Net effect: incremental behavior is now consistent in both collection and 
post-collection processing, with major runtime reduction on previously 
problematic projects.
   
   ### Does this close any open issues?
   No
   
   ### Screenshots
   <img width="1284" height="594" alt="image" 
src="https://github.com/user-attachments/assets/324c4839-afdd-470b-8497-008ba0a85f05";
 />
   
   
   ### Other Information
   Any other information that is important to this PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to