gaogaotiantian commented on PR #57587: URL: https://github.com/apache/spark/pull/57587#issuecomment-5248590111
> First, regarding the new Grimp dependency: how reliable and accurate is its import analysis for a project like PySpark? Do we know of other large projects using it in CI for test selection? It would be useful to understand its maintenance record, compatibility guarantees, and real-world adoption before adding it as a dependency. `grimp` is a relatively mature library. It is still maintained actively - https://github.com/python-grimp/grimp . Daily download from pypistats is a bout 600k https://pypistats.org/packages/grimp . I don't think it's that difficult to analyze import graph for python. It won't be 100% accurate because of the dynamic nature of Python, but it should be pretty close. We have a lot of dependencies that are not maintained at all, some are even used by pyspark directly (`memory_profiler` for example). I think as a pure development dependency, `grimp` is in a good state now. We can throw it away in the future too, should not be too hard. > Second, I am fine with using smart selection in post-merge CI to save resources. However, pre-merge CI is the merge gate, so we should enable it there only when we are confident that the analysis cannot skip an impacted test. In particular, we need to account for dynamic imports and other indirect dependencies, and fall back to running tests whenever the result is uncertain. This is intended to be a post-merge only optimization. I did not make it so in the first attempt. I updated the code so the smart test selection only runs for push actions on apache/spark. pre-merge CIs are not affected. This should also relax the concern for the first question. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
