On Sun, 26 Oct 2025 at 01:06, '[email protected]' via sympy <[email protected]> wrote: > > Here's a brainstorming idea for how to implement something to address your > valid concerns. > How about the following policy? > No review of a pull request will occur unless it meets certain minimum > requirements: > 1) It passes all pre-existing tests; > 2) It includes test coverage for all new code; > 3) It includes tests covering any bug fixes.
I think that what will happen is that the author will pass these instructions to the LLM agent and then the agent will generate some code that superficially resembles meeting these criteria. Then the PR description will have a bunch of emoji-filled bullet points redundantly stating that it meets those criteria. I'm not going to point to the specific PR but I closed one where the description had a statement in it like "You can run the tests with `pytest sympy/foo/bar`" That is literally an instruction from the LLM to the user for how they can test the generated code and if you actually run the test command it clearly shows that the code doesn't work. It was still submitted in that form as a spam PR though. Of course the tests in CI did not pass and it is not hard to see the problem in that case but other cases can be more subtle than this. It is not hard to generate code that passes all existing tests, includes coverage etc while still being gibberish and this is really the problem with using LLMs. There isn't any substitute in this situation for actual humans doing real thinking. -- Oscar -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/sympy/CAHVvXxRbOp%3DaScmgT8uTzj2VQcDV2XseqN%3Dy_GHYN3yCXW_uGA%40mail.gmail.com.
