I discuss three issues in the Project Glasswing initial update of May 22 in this message.
Reference: https://www.anthropic.com/research/glasswing-initial-update 1. Disagreement in severity assessment between Anthropic and maintainer The chart has boxes on the bottom line which say this: 1,586 findings Reported to maintainers 1,451 findings Acknowledged by maintainer The text does not give the number of findings acknowledged by maintainer. While the text discusses only high and critical vulnerabilities the chart lumps all levels together. There must surely be cases in which Mythos and the Anthropic vetting staff classify an issue as high severity while the maintainer considers it low severity. Do such cases count as "acknowledged by maintainer"? We can't tell. Many observers say that it is in Anthropic's interest to claim that the problems Mythos is finding are severe. However in some instances the maintainer might be underestimating the severity. Claude Mythos is good at chaining together vulnerabilities to create an exploit. An issue which appears minor in itself may be combined with others to create a serious security hole. 2. Pareto rule The Anthropic team has discovered "3,900 high-or critical-severity vulnerabilities in open-source code". 1000 projects were scanned. We should not expect the vulnerabilities to be evenly distributed. Applying the 80:20 rule we get 3120 in 200 projects and the remaining 780 in 800 projects. 3. Quality of code written by "AI" The security-related findings by Claude Mythos seem to show that "AI" is getting better at writing code. On the contrary we have reports like this which claim that code produced by "AI" often introduces vulnerabilities: Report finds AI-generated code poses security risks July 30, 2025 https://www.eenewseurope.com/en/report-finds-ai-generated-code-poses-security-risks/ Veracode has unveiled its 2025 GenAI Code Security Report, revealing critical security flaws in AI-generated code. The study analysed 80 curated coding tasks across more than 100 large language models (LLMs), revealing that while AI produces functional code, it introduces security vulnerabilities in 45 per cent of cases. The research shows that despite advances in AI-generated code and the ability of LLMs to generate syntactically correct code, security performance has not kept up, remaining unchanged over time. Another concerning trend is that when presented with a choice between secure and insecure coding methods, GenAI models opted for the insecure option 45 per cent of the time. Finding a large number of vulnerabilities is not the same as a guarantee that all vulnerabilities will be found. "AI" can write code and scan code for problems but that gives us no guarantee that it will be always aware of problems in its output or input. Some people are interested in this. They feed old versions of a program with a well-known issue and test whether the LLM can actually find it. There is no mention of tests of this kind by Anthropic in the initial report. Akira Urushibata _______________________________________________ libreplanet-discuss mailing list [email protected] https://lists.libreplanet.org/mailman/listinfo/libreplanet-discuss
