Agree, flaky tests are pretty annoying. I'll try to watch more carefully new "embedded data-source" issues (fongo, ES, cassandra). They introduced more "non-determinism" because they now run as part of regular build which means executed much more often than IT.
Last commit was for a more deterministic issue (incorrect parsing of new java version format <http://openjdk.java.net/jeps/223>). On Mon, Jul 30, 2018 at 12:58 PM Julian Hyde <[email protected]> wrote: > On balance, I don’t think we should back out CassandraAdapterTest. But we > do need to continue working to make it more resilient. Any test that > generates too many false negatives over the long run should be disabled, > and this is no exception. > > And as the other test results show, it’s not the only flaky part of the > test suite, when the machine is stressed. One thing we can do is to avoid > timeouts whenever possible. > > Julian > > > > On Jul 30, 2018, at 9:30 AM, Andrei Sereda <[email protected]> wrote: > > > > Pls check and confirm that the following PR fixes the issue: > > https://github.com/apache/calcite/pull/770 > > It addresses build failures due to version parsing. > > > > Stale folders / files will be addressed in a different PR. > > > > > > On Mon, Jul 30, 2018 at 10:10 AM Andrei Sereda <[email protected]> wrote: > > > >> This must be something specific to Cassandra Unit. Will check > >> > >> On Mon, Jul 30, 2018, 08:59 Sergey Nuyanzin <[email protected]> > wrote: > >> > >>> There is one more strange thing (at least on Windows): while building a > >>> file with name ".toDelete" is generated under calcite\cassandra and > it's > >>> not removed by the end of tests > >>> Is there a way to make cassandra generates these files in target > directory > >>> e.g.? > >>> > >>> > >>> > >>> On Mon, Jul 30, 2018 at 3:52 PM, Andrei Sereda <[email protected]> > wrote: > >>> > >>>> Most of the problems are during test init Phase. Most likely with > >>> version > >>>> string (eg. 11-ea for JDK11). > >>>> I'll fix that. > >>>> > >>>> > >>>> On Mon, Jul 30, 2018 at 8:16 AM Michael Mior <[email protected]> > wrote: > >>>> > >>>>> I tested myself a fair bit under Ubuntu before pushing this and > didn't > >>>> see > >>>>> any of these issues myself. That said, I agree that it's important > the > >>>> test > >>>>> suite be stable. I'm fine with reverting for now or (more preferably > >>> IMO) > >>>>> just disabling these tests by default. > >>>>> > >>>>> -- > >>>>> Michael Mior > >>>>> [email protected] > >>>>> > >>>>> > >>>>> > >>>>> Le lun. 30 juil. 2018 à 00:59, Julian Hyde <[email protected]> a > >>> écrit : > >>>>> > >>>>>> I ran on several JDK versions, all on Ubuntu Linux. The machine was > >>>>>> fairly heavily loaded (I was copying one filesystem to another at > >>> the > >>>>>> time). > >>>>>> > >>>>>> Here's the error from OpenJDK10: > >>>>>> > >>>>>> [INFO] Running org.apache.calcite.test.CassandraAdapterTest > >>>>>> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time > >>>>>> elapsed: 0.001 s <<< FAILURE! - in > >>>>>> org.apache.calcite.test.CassandraAdapterTest > >>>>>> [ERROR] org.apache.calcite.test.CassandraAdapterTest Time elapsed: > >>>>>> 0.001 s <<< ERROR! > >>>>>> java.lang.ExceptionInInitializerError > >>>>>> at > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.initCassandraIfEnabled( > >>>> CassandraAdapterTest.java:106) > >>>>>> at > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.<clinit>( > >>>> CassandraAdapterTest.java:56) > >>>>>> Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end > >>> -1, > >>>>>> length 5 > >>>>>> at > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.initCassandraIfEnabled( > >>>> CassandraAdapterTest.java:106) > >>>>>> at > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.<clinit>( > >>>> CassandraAdapterTest.java:56) > >>>>>> > >>>>>> Here's the error in JDK 11: > >>>>>> > >>>>>> [INFO] Running org.apache.calcite.test.CassandraAdapterTest > >>>>>> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time > >>>>>> elapsed: 0 s <<< FAILURE! - in > >>>>>> org.apache.calcite.test.CassandraAdapterTest > >>>>>> [ERROR] org.apache.calcite.test.CassandraAdapterTest Time elapsed: > >>> 0 > >>>>>> s <<< ERROR! > >>>>>> java.lang.ExceptionInInitializerError > >>>>>> at > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.enabled( > >>>> CassandraAdapterTest.java:81) > >>>>>> at > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.initCassandraIfEnabled( > >>>> CassandraAdapterTest.java:88) > >>>>>> at > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.<clinit>( > >>>> CassandraAdapterTest.java:56) > >>>>>> > >>>>>> Here's the failure from JDK 9: > >>>>>> > >>>>>> [INFO] Running org.apache.calcite.test.CassandraAdapterTest > >>>>>> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time > >>>>>> elapsed: 0.001 s <<< FAILURE! - in > >>>>>> org.apache.calcite.test.CassandraAdapterTest > >>>>>> [ERROR] org.apache.calcite.test.CassandraAdapterTest Time elapsed: > >>>>>> 0.001 s <<< FAILURE! > >>>>>> java.lang.AssertionError: Cassandra daemon did not start within > >>> timeout > >>>>>> > >>>>>> Here's the failure from JDK 10: > >>>>>> > >>>>>> [INFO] Running org.apache.calcite.jdbc.CalciteRemoteDriverTest > >>>>>> [ERROR] Tests run: 28, Failures: 1, Errors: 0, Skipped: 0, Time > >>>>>> elapsed: 0.642 s <<< FAILURE! - in > >>>>>> org.apache.calcite.jdbc.CalciteRemoteDriverTest > >>>>>> [ERROR] > >>>>>> testRemoteExecuteQuery(org.apache.calcite.jdbc. > >>>> CalciteRemoteDriverTest) > >>>>>> Time elapsed: 0.047 s <<< FAILURE! > >>>>>> java.lang.AssertionError: > >>>>>> > >>>>>> Expected: "EXPR$0=1; EXPR$1=a\nEXPR$0=null; EXPR$1=b" > >>>>>> but: was "" > >>>>>> at > >>>>>> > >>>>> > >>> org.apache.calcite.jdbc.CalciteRemoteDriverTest.testRemoteExecuteQuery( > >>>> CalciteRemoteDriverTest.java:280) > >>>>>> > >>>>>> In conclusion: It's worrying that the suite shows 4 different cracks > >>>>>> under 4 different JDKs. Clearly the load on my machine was making > >>>>>> problems worse, and granted, the problems are just testing problems, > >>>>>> not real bugs. But flaky test suites waste time and effort. There > >>> are > >>>>>> indications that the new embedded Cassandra test is more flaky than > >>>>>> most. > >>>>>> > >>>>>> Julian > >>>>>> > >>>>>> > >>>>>> On Sun, Jul 29, 2018 at 11:33 AM, Andrei Sereda <[email protected]> > >>>>> wrote: > >>>>>>> What version of java / OS / maven do you have ? What is your maven > >>>>>> command > >>>>>>> ? > >>>>>>> > >>>>>>> Things pass for me on MacOS X java 8,9 and 10. > >>>>>>> > >>>>>>> On Sun, Jul 29, 2018 at 2:05 PM Julian Hyde <[email protected]> > >>>> wrote: > >>>>>>> > >>>>>>>> I'm seeing the following error when I run the tests on bd0e14002 > >>>>>>>> origin/master. Anyone else see it? > >>>>>>>> > >>>>>>>> [INFO] Running org.apache.calcite.test.CassandraAdapterTest > >>>>>>>> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time > >>>>>>>> elapsed: 0.001 s <<< FAILURE! - in > >>>>>>>> org.apache.calcite.test.CassandraAdapterTest > >>>>>>>> [ERROR] org.apache.calcite.test.CassandraAdapterTest Time > >>> elapsed: > >>>>>>>> 0.001 s <<< ERROR! > >>>>>>>> java.lang.ExceptionInInitializerError > >>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.initCassandraIfEnabled( > >>>> CassandraAdapterTest.java:106) > >>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.<clinit>( > >>>> CassandraAdapterTest.java:56) > >>>>>>>> Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, > >>> end > >>>> -1, > >>>>>>>> length 5 > >>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.initCassandraIfEnabled( > >>>> CassandraAdapterTest.java:106) > >>>>>>>> at > >>>>>>>> > >>>>>> > >>>>> org.apache.calcite.test.CassandraAdapterTest.<clinit>( > >>>> CassandraAdapterTest.java:56) > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> Best regards, > >>> Sergey > >>> > >> > >
