Interesting numbers, to say the least :) Separating test suites, as suggested by Vladimir sounds good.
Probably off-placed question: As Ignite provides multiple features and is suitable for multiple use case- should we categorize test suites based on features? ------ Rohit Mohta > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <[email protected]> wrote: > > In addition to optimizing the test suites, should we add more TC agents as > well? > > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <[email protected]> > wrote: > >> And one more thing - how much time is spent on duplicated "Binary" suites? >> *15 >> hours*, or almost *40%* of total "Run All" execution time. It means, that >> if move suites for *OptimizedMarshaller *(which we consider deprecated) >> into a separate circuit (which will be executed not per-branch, but e.g. >> nightly) we will almost double peak TeamCity throughput. >> >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <[email protected]> >> wrote: >> >>> Igniters, >>> >>> I collected several thought-provoking numbers for our TeamCity >>> (config-variations tests are excluded): >>> >>> * Number of Java suites: 78 >>> * Total execution time without build overhead: 38.5 hours >>> * Total test classes: 3376 >>> * Total tests: 60681 >>> >>> * 25% of time is consumed by 881 tests (41 test class) >>> * 50% of time is consumed by 2755 tests (153 test class) >>> * 75% of time is consumed by 15911 tests (443 test class) >>> >>> Or to be more demonstrative: >>> * 1.5% of tests consume 25% of time >>> * 4.5% of tests consume 50% of time >>> * 26.2% tests consume 75% of time >>> >>> TOP 3 test classes: >>> * CacheSerializableTransactionsTest - 46 minutes >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes >>> * GridCachePartitionedNodeRestartTest - 25 minutes >>> >>> Morale: >>> 1) We have lots of poor failover/concurrency tests requiring >>> disproportional time to run; >>> 2) If we split all tests into "core" and "failover/concurrency" parts >>> properly, then *core* part will take *~15 mins* to answer you with high >>> level of confidence whether your change broken anything or not. >>> >>> Thoughts? >>> >>> Vladimir. >>> >>
