In addition to optimizing the test suites, should we add more TC agents as well?
On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <[email protected]> wrote: > And one more thing - how much time is spent on duplicated "Binary" suites? > *15 > hours*, or almost *40%* of total "Run All" execution time. It means, that > if move suites for *OptimizedMarshaller *(which we consider deprecated) > into a separate circuit (which will be executed not per-branch, but e.g. > nightly) we will almost double peak TeamCity throughput. > > On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <[email protected]> > wrote: > > > Igniters, > > > > I collected several thought-provoking numbers for our TeamCity > > (config-variations tests are excluded): > > > > * Number of Java suites: 78 > > * Total execution time without build overhead: 38.5 hours > > * Total test classes: 3376 > > * Total tests: 60681 > > > > * 25% of time is consumed by 881 tests (41 test class) > > * 50% of time is consumed by 2755 tests (153 test class) > > * 75% of time is consumed by 15911 tests (443 test class) > > > > Or to be more demonstrative: > > * 1.5% of tests consume 25% of time > > * 4.5% of tests consume 50% of time > > * 26.2% tests consume 75% of time > > > > TOP 3 test classes: > > * CacheSerializableTransactionsTest - 46 minutes > > * IgniteCacheCrossCacheTxFailoverTest - 30 minutes > > * GridCachePartitionedNodeRestartTest - 25 minutes > > > > Morale: > > 1) We have lots of poor failover/concurrency tests requiring > > disproportional time to run; > > 2) If we split all tests into "core" and "failover/concurrency" parts > > properly, then *core* part will take *~15 mins* to answer you with high > > level of confidence whether your change broken anything or not. > > > > Thoughts? > > > > Vladimir. > > >
