gustavobini commented on issue #7516: URL: https://github.com/apache/incubator-devlake/issues/7516#issuecomment-2132780921
https://github.com/apache/incubator-devlake/blob/6dca288d5e33bb55a8b98995df97ead71b76a082/backend/plugins/jenkins/tasks/build_collector.go#L143 does not collect data incrementally. It could use `NewStatefulApiCollectorForFinalizableEntity` but that does not support iterators. What about changing `collectMultiBranchJobApiBuilds` to call https://github.com/apache/incubator-devlake/blob/6dca288d5e33bb55a8b98995df97ead71b76a082/backend/plugins/jenkins/tasks/build_collector.go#L72 for each entry in the iterator? This could help, but https://github.com/apache/incubator-devlake/blob/3ae23bfdda0537e3cbe5eed17340e4fb115af1ce/backend/plugins/jenkins/tasks/build_extractor.go#L40 will still select all data from the raw table, including processed data. I don't know if raw tables are expected to have that data forever, or if they should be cleaned every now and then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org