Just so we are clear, I am opposed to using AI to automatically open pull requests in this way, and I would ask that people please don't do this in the SymPy repository.
I think that AI tools can be useful to help find and fix bugs, and if that can be streamlined more, for instance, by making the AI automatically read the issue, that is great. But the thing that shouldn't be automated is the actual opening of the pull request. There still needs to be a human in the process who actually looks at the code and can take responsibility for it. The accuracy of these tools is not high enough yet that the whole process should be done automatically. All this does is waste maintainer time. Aaron Meurer On Wed, Jul 24, 2024 at 10:43 AM Sangyub Lee <[email protected]> wrote: > > I notice a PR created today that is suspected to be generated by LLM (sorry > if it is false accusation) > Title: Fix is_zero Attribute Handling in Expression SimplificationDes… by > Devansh-46 · Pull Request #26850 · sympy/sympy (github.com) > However, it doesn't turn out to be successful anyway, due to the failing of > tests, and coming up with quite terse but nonsensical description of PR. > > Although the SWE paper notes about significant improvements over ChatGPT or > old models about performance, > I still have a lot skepticism that the LLM models can write the code or fix > issues of stuff that may be poorly organized, and may need many contextual > understanding, like simplify function. > And it may be more easier for LLM to fix issues of code that is easy to > understand and structured well by experienced programmars, so it won't easily > degrade the jobs of good programmers. > However, it may be difficult to experience the improvement of performance, > because they are still in research status and not used in production like > Copilot. > On Sunday, May 19, 2024 at 2:41:24 AM UTC+2 [email protected] wrote: >> >> The SWE-agent project uses LLMs to try to automatically fix issues in >> GitHub repositories. I found their paper interesting, mostly because >> they make extensive use of SymPy as a test repository. >> https://swe-agent.com/ >> >> Apparently there are quite a few SymPy issues in the SWE-bench >> dataset, which is a dataset of issues and corresponding pull requests >> in open source projects. https://www.swebench.com/ >> >> Their model was able to fix 10% of SymPy issues in the dataset. That's >> obviously not going to replace human developers any time soon, but >> it's still interesting. From what I could tell, the issues seem to be >> biased towards more easy/straightforward ones (i.e., ones that are >> easy to verify if a fix is correct or not). But still, if LLMs are >> reaching a point where they can fix bugs completely automatically that >> could be very useful. >> >> Aaron Meurer > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/d93b3572-383b-4465-9172-790de55d8b8dn%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6LctDsig_FJ5FucV8S9AyHSnrf0Vc8P8EJCZM9K%3DQFb0w%40mail.gmail.com.
