Dima, I do not think we really need more hardware for now. 40 min test will not run faster if we add more agents. We have a problem with tests quality, not quantity.
Rohit, Yes, Ignite is ver advanced framework, so we have about 80-90 different suites, testing different parts of API. Vladimir. On Sat, Dec 17, 2016 at 2:47 AM, Rohit Mohta <[email protected]> wrote: > Interesting numbers, to say the least :) > > Separating test suites, as suggested by Vladimir sounds good. > > Probably off-placed question: > As Ignite provides multiple features and is suitable for multiple use > case- should we categorize test suites based on features? > > ------ > Rohit Mohta > > > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <[email protected]> > wrote: > > > > In addition to optimizing the test suites, should we add more TC agents > as > > well? > > > > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <[email protected]> > > wrote: > > > >> And one more thing - how much time is spent on duplicated "Binary" > suites? > >> *15 > >> hours*, or almost *40%* of total "Run All" execution time. It means, > that > >> if move suites for *OptimizedMarshaller *(which we consider deprecated) > >> into a separate circuit (which will be executed not per-branch, but e.g. > >> nightly) we will almost double peak TeamCity throughput. > >> > >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <[email protected]> > >> wrote: > >> > >>> Igniters, > >>> > >>> I collected several thought-provoking numbers for our TeamCity > >>> (config-variations tests are excluded): > >>> > >>> * Number of Java suites: 78 > >>> * Total execution time without build overhead: 38.5 hours > >>> * Total test classes: 3376 > >>> * Total tests: 60681 > >>> > >>> * 25% of time is consumed by 881 tests (41 test class) > >>> * 50% of time is consumed by 2755 tests (153 test class) > >>> * 75% of time is consumed by 15911 tests (443 test class) > >>> > >>> Or to be more demonstrative: > >>> * 1.5% of tests consume 25% of time > >>> * 4.5% of tests consume 50% of time > >>> * 26.2% tests consume 75% of time > >>> > >>> TOP 3 test classes: > >>> * CacheSerializableTransactionsTest - 46 minutes > >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes > >>> * GridCachePartitionedNodeRestartTest - 25 minutes > >>> > >>> Morale: > >>> 1) We have lots of poor failover/concurrency tests requiring > >>> disproportional time to run; > >>> 2) If we split all tests into "core" and "failover/concurrency" parts > >>> properly, then *core* part will take *~15 mins* to answer you with high > >>> level of confidence whether your change broken anything or not. > >>> > >>> Thoughts? > >>> > >>> Vladimir. > >>> > >> >
