On 02/11/2026 13:57, Rémy Maucherat via dev wrote:
On Mon, Nov 2, 2026 at 2:27 PM Dimitris Soumis <[email protected]> wrote:
On Mon, Nov 2, 2026 at 2:24 PM Rémy Maucherat via dev <[email protected]>
wrote:
On Mon, Nov 2, 2026 at 12:41 PM Mark Thomas via dev
<[email protected]> wrote:
All,
The start of the month is here so we should be thinking about tagging
the November releases.
The various AI reviews have resulted in significantly higher numbers of
commits and PRs. There are also a handful of open bugs.
Normally, I'd aim to review all of the commits, resolve the new PRs and
the open bugs before release.
Currently, I am 2-3 weeks behind and I am unsure what to do.
Any BZ issue or PR that relates to an enhancement can wait until the
next release but that still needs those PRs and bugs to be triaged. The
commit reviews are harder. I was hoping that I'd be able to work through
them fairly quickly but that isn't the case. I am finding additional
fixes that are required or regressions quite frequently. Many of these
could be classes as no worse than the original code but there are
occasionally more serious regressions.
Yes, the extra fixes are very useful. It would be better if they were
not needed of course (working on that ...).
My reviews have speed up considerably this morning. There is a clear
correlation between code complexity and speed of review / further fixes
being required. That suggests to me that we need to be using more
powerful models for the more complex code. Maybe look at using the free
AI credits for those models that have been offered to ASF committers?
To me, it seems clear that the whole development process has changed.
Therefore, I suggest changing the review process to introduce a risk-based
approach. For committed reviews, maybe focus review on potential high-impact
areas and limit nonessential changes ~1 week before tagging.
I agree the development process is changing. We can't not review code
and I've seen enough obscure bugs in what I thought was non-critical
code lead to a vulnerability that I'd be very cautious of reducing the
level of review to any code.
My current thinking is using more powerful models will enable the
initial fix to be cleaner. That addresses the time problem by reducing
the time needed for the subsequent review.
If we wanted to take it one step further, it might be worth considering
altering the release cycle from monthly to something more frequent so reported
regressions can be fixed much faster. To reduce the overhead of this, we could
also explore utilizing the Apache Trusted Releases project.
Increasing the release rate to address an increased volume of
regressions seems like the wrong approach to me. I'd rather focus on
reducing the regressions (see above).
I'm also not sure our user base could keep up. I'm fairly sure only a
very small percentage of our users are keeping up with the latest
releases as it is. I agree that the way things are going they do need to
do a better job of keeping up.
AI is accelerating development, and I think our processes should evolve to
accommodate that.
It is still relatively early days with AI. At the moment, I think we are
applying AI inconsistently across the development process so it is
speeding up some areas and creating bottlenecks in others. I think we
need to apply a little more AI upfront - particularly to the changes to
more complex areas - to reduce the time spent on reviews which is where
the bottleneck is currently.
LLMs seem to be making things different overall (for the worse, of
course, given the volume of very very hard to find CVEs that would
have never been found without it).
Code reviews produce (lots of) defects (PRs are now all produced by
LLMs and are basically the same thing). These defects are really non
critical: we know Tomcat is stable and worked fine 6 months ago. So
some edge cases nobody ever cared or ran into are getting fixed,
because once we know about them, we have to fix them. However,
individually, the value for each seems infinitesimal. Only if
everything gets fixed, we could assume a marginal reliability
improvement (not to be underestimated either since the last bit of
reliability requires a maximum amount of effort to achieve).
Agreed. I do wonder if some the claimed edge cases are even possible at
all but in most cases it takes less time to fix the edge case that it
does to reason it doesn't apply.
We also have new features that are possible. Nobody would have ever
cared about doing a Manager 2.0 webapp, well, LLMs do a decent job at
that. This seems useful to users (webapp is "better" than the existing
one), but the benefit is also slim.
We can also do features that nobody really wants (QUIC). Similar low benefit.
I would argue that features like TLS support for clustering do provide
real benefit and there is no way I could have implemented that in the
time I did without AI support.
I did briefly start to look at using AI to do refactoring of some of the
more complex code to simplify it but ran out of time before I made any
real progress. I hope that this is an area that we will be able to come
back to.
Overall: not great, but I feel like I have to do it.
Ack. I feel the same way.
Rémy
Do we tag anyway and address regressions as they are reported?
+1
I'm not a huge fan of this approach but I recognise we might need to do
this.
Is there a better way to triage a large number of commits for closer review?
Does delaying the tag gain us anything?
The [VOTE] was posted on the 6th last month, so there's time.
With the speed I am currently working through the reviews, I am a lot
more hopeful. I am ignoring the jdbc-pool changes for now on the grounds
that DBCP is the default and users that find a regression in jdbc-pool
can always use DBCP (or a jdbc-pool JAR from a previous Tomcat version).
There are 5 BZs to look at right now, so it's not that bad. To be
honest, even AI centric projects have given up long ago.
Note: the BZ "Long Format" button allows saving the html and
triaging/reproducing issues fast (usually at least). I used it to
triage the jdbc-pool BZs.
Good to know.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
