On Tue, Oct 7, 2025 at 9:32 PM Coty Sutherland <[email protected]> wrote: > > Hi all, > > Before I started implementation and submitted a PR, I wanted to share a > proposal for some new test targets in the build.xml. The idea is to > introduce structured test categories to improve developer productivity and > CI efficiency through faster, more targeted test execution without the need > to know or use the various fileset patterns. > > *New Test Targets*
I think that's a good idea, but the times are relative. For example on GitHub CI (we're not paying ;) ) it takes a bit more than what it really should. So I'm a bit worried that if you start talking about times the base "smoke" cuts down too many tests. > * ant smoke-test - Runs fast smoke tests (~30 seconds) that verify basic > functionality across all major Tomcat components including server startup, > core engine, etc. Tests essential class loading and API availability. The coverage might be significantly lower here. > * ant test-quick - Runs unit tests and critical integration tests (~5 > minutes) for development validation. Excludes integration scenarios, > performance tests, and complex deployment tests. I think the GH CI suite is reasonable in terms of coverage. It takes 12 minutes there (and much less elsewhere). I would not cut additional tests for this one. Otherwise the TLS and many "hard" (?) tests have to be cut and that's likely bad. > * ant test-components - Runs full component testing (~20 minutes) with unit > tests for specific components in parallel (6 components; each with their > own test target). Excludes cross-component integration tests. > * ant test-integration - Runs cross-component integration tests (~30 > minutes) including WebSocket, SSL/TLS, clustering, session management, > authentication, valves, filters, startup lifecycle, and JSP-servlet > integration. I'm not sure we have significant differences here right now between these two definitions. Note: The full testsuite is twice as fast with 12 compared to 11 or 10.1, since no NIO2. Similarly, Tomcat 9 also has APR, so about 3x longer than 12. Adding a "full" run could make a significant difference there ! Note 2: The TLS tests also need JSSE + FFM + tomcat-native to count as a full run. > * ant test-performance - Runs performance tests for benchmarking and > optimization, including timing-sensitive code paths, memory usage, and > throughput tests. Isolated from other categories to prevent flaky failures. > * ant test-tribes-system - Runs comprehensive clustering system tests (30+ > minutes) for the Tribes clustering component. These are high-resource, > long-running integration scenarios that were previously excluded from the > main test suite due to regular failures, now available for thorough > validation when working on clustering functionality. So indeed there are the specific performance tests, and maybe some others from tribes (never tried ...). > Note: times mentioned above are guesstimates based on running with a few > test threads since I haven't implemented anything yet. > > *Component-Specific Test Targets* > > * ant test-component-catalina - Runs all Catalina tests > * ant test-component-coyote - Runs all Coyote tests > * ant test-component-jasper - Runs all JSP tests > * ant test-component-el - Runs all EL tests > * ant test-component-tomcat - Runs all Tomcat utilities, WebSocket, > logging, and JNDI tests > * ant test-component-servlet - Runs all servlet tests > > *Key Benefits* > > For Developers: > - 30-second smoke tests feedback vs 15-30 minute full suite (depending on > available test threads) > - Run only relevant component tests for the systems you're working on > - More obvious test targets with specific purpose > - Quick validation with a shorter feedback loop before commits > > For CI: > - Shorter test runs across multiple jobs/platforms to reduce costs (if > there were any) > - Replace 10-20 minute "smoketest" with 30-second validation (or 5 min > quick tests) for faster builds and a notable decrease in compute time used > for every commit > - Avoids the need to update the ci.yml to exclude new tests that may cause > longer runtimes > > *Implementation* > > The implementation of this plan would follow existing conventions to > utilize the same JVM args, properties, and exclude patterns as current > runtests macro preserving compatibility with the current test workflows. > There wouldn't be any change to existing test targets, only the new ones > introduced. The only new change to the suite would be the addition of the > SmokeTest designation in filenames if we wanted to include new tests for > that target; everything else is just creating targets from existing > filesets/patterns. > > Thoughts? If there aren't any objections I'll start working on a PR :D +1 Rémy --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
