Agree with Vladimir's suggestions. Great analysis! --Yakov
2016-12-17 11:36 GMT+03:00 Vladimir Ozerov <[email protected]>: > Dima, > I do not think we really need more hardware for now. 40 min test will not > run faster if we add more agents. We have a problem with tests quality, not > quantity. > > Rohit, > Yes, Ignite is ver advanced framework, so we have about 80-90 different > suites, testing different parts of API. > > Vladimir. > > On Sat, Dec 17, 2016 at 2:47 AM, Rohit Mohta <[email protected]> > wrote: > > > Interesting numbers, to say the least :) > > > > Separating test suites, as suggested by Vladimir sounds good. > > > > Probably off-placed question: > > As Ignite provides multiple features and is suitable for multiple use > > case- should we categorize test suites based on features? > > > > ------ > > Rohit Mohta > > > > > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <[email protected]> > > wrote: > > > > > > In addition to optimizing the test suites, should we add more TC agents > > as > > > well? > > > > > > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <[email protected] > > > > > wrote: > > > > > >> And one more thing - how much time is spent on duplicated "Binary" > > suites? > > >> *15 > > >> hours*, or almost *40%* of total "Run All" execution time. It means, > > that > > >> if move suites for *OptimizedMarshaller *(which we consider > deprecated) > > >> into a separate circuit (which will be executed not per-branch, but > e.g. > > >> nightly) we will almost double peak TeamCity throughput. > > >> > > >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov < > [email protected]> > > >> wrote: > > >> > > >>> Igniters, > > >>> > > >>> I collected several thought-provoking numbers for our TeamCity > > >>> (config-variations tests are excluded): > > >>> > > >>> * Number of Java suites: 78 > > >>> * Total execution time without build overhead: 38.5 hours > > >>> * Total test classes: 3376 > > >>> * Total tests: 60681 > > >>> > > >>> * 25% of time is consumed by 881 tests (41 test class) > > >>> * 50% of time is consumed by 2755 tests (153 test class) > > >>> * 75% of time is consumed by 15911 tests (443 test class) > > >>> > > >>> Or to be more demonstrative: > > >>> * 1.5% of tests consume 25% of time > > >>> * 4.5% of tests consume 50% of time > > >>> * 26.2% tests consume 75% of time > > >>> > > >>> TOP 3 test classes: > > >>> * CacheSerializableTransactionsTest - 46 minutes > > >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes > > >>> * GridCachePartitionedNodeRestartTest - 25 minutes > > >>> > > >>> Morale: > > >>> 1) We have lots of poor failover/concurrency tests requiring > > >>> disproportional time to run; > > >>> 2) If we split all tests into "core" and "failover/concurrency" parts > > >>> properly, then *core* part will take *~15 mins* to answer you with > high > > >>> level of confidence whether your change broken anything or not. > > >>> > > >>> Thoughts? > > >>> > > >>> Vladimir. > > >>> > > >> > > >
