gaogaotiantian commented on PR #57461: URL: https://github.com/apache/spark/pull/57461#issuecomment-5095673421
I have a lot of doubts for this approach in general. * Is copilot free? Will it be free forever? How much we need to pay for AI to select tests for us, comparing to the money we save for running those tests? We are running this on Apache organization so it's probably different than normal public personal repo. * How accurate is the output? How reliable it is? * Is it rational to have a hard limit for tests to run? There will be changes that touch the very core of the system and we probably want to run full. TBH I don't think this is the way we should go at this point. If we use 2500k min vs 250k min limit, we probably want AI to select tests for us because there's no way we can save it properly. We have to compromise on coverage. However, it's like 350k vs 250k now in worst weeks. I think we should stick to rule based approach first. We can have "smart selection", but I have serious doubts if we just ask copilot to do it. We should be at least 99.9% sure that the test is irrelevant, rather than blindly trust AI on it. I believe we still have a lot of space to improve in "traditional programming" field. For example, we never really try to improve the speed of Java tests. And now our proposal is just to "not run some of them". We should clean up some tests (especially slow ones) that do not give us extra coverage. We should probably make it possible for developers to manually skip the post-merge CI in merge script (by adding a special tag in commit message like `SKIP_TEST`). Sometimes we know the change is trivial and does not have impact to the actual CI, but our script is not smart enough to pick it up. This would be traceable as the commit message will be there and the committer is responsible for it. As for smart selection, I would start a project soon to analyze the import graph for pyspark so less tests run if just a small part of the code/test is changed. I don't know how we deal with Java. TBH I think Java tests take most of the time now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
