Hi all, I wonder what is a maximum duration for newly added test case that will still be "reasonable". I mean a test case that is executed in Travis to validate pull requests, and a test case that is executed as a part of regular `mvn test` / `mvn install`.
Could you please pick maximum test duration that would still be reasonable from your point of view? I would be glad to hear explanations as well. 1sec 5sec <-- I think this is the maximum reasonable test duration (for a single test) 10sec 30sec 60sec (1m) 120sec (2m) 240sec (4m) 480sec (8m) 960sec (16m) 1920sec (32m) Note: the duration might depend on CPU performance, however could we just ignore that? For instance: suppose I implement a fix for "CALCITE-2506 RexSimplify: coalesce(unaryPlus(nullInt), unaryPlus(vInt())) results in AssertionError: result mismatch", and I add a test case that takes 60 seconds to execute. Is that reasonable? Then I fix "CALCITE-2527 Simplify OR with comparisons leads to AssertionError: result mismatch" and add a test case that takes yet another 60 seconds. Then I fix "CALCITE-2526 Simplify OR with comparisons leads to NPE" and add a test that takes yet another 60 seconds. As for me, 60 seconds for a single test is unreasonable since it would easily bump test execution times to heaven, especially in Travis that uses one thread only. Note: there are cases like "it takes 20-30 seconds to start Cassandra", however I assume Cassandra is started only once, then all the relevant tests are run. Then "restart Cassandra for each test method" is unreasonable, while starting it once is fine. Vladimir
