Hi Michael, I believe the problem can be not related to RAM space, since today I run into the same issue on 200GB RAM machine.
I will try to run intensive/non-intensive tests separately. Thank you, Sergey From: Michael Shuler Sent: Thursday, October 19, 2017 3:58 PM To: dev@cassandra.apache.org Subject: Re: Cassandra DTests deadlocks on random test 7.6G RAM may be a little bit too small, we've seen similar random hangs in the past on non-resource-intensive tests on m3.large. It doesn't appear you are skipping resource-intensive tests. Our standard dtest instance type has been an m3.xlarge, and the resource-intensive tests are run m3.2xlarge. (or something comparable on RAM & SSD) dtest run command (excludes resource-intensive tests): https://github.com/apache/cassandra-builds/blob/master/build-scripts/cassandra-dtest.sh#L53 dtest-large run command (only resource-intensive tests): https://github.com/apache/cassandra-builds/blob/master/build-scripts/cassandra-dtest.sh#L61 Running dtest without excluding resource-intensive will run everything. If you wish to troubleshoot your tests when they hang, there should be /tmp/dtest-XXXXX directories with the ccm cluster left on disk from the hung test, since they never get to cleanup stage. -- Michael On 10/19/2017 05:29 AM, Sergey La wrote: > Hi! > > I have created the patch for the Cassandra version 3.0.14 and trying to > test it using the cassandra dtests. > > Problem is - dtests deadlocks at some random tests, time and again - on > unpatched 3.0.14 version of Cassandra. > > What I have done. > > I cloned the cassandra repository (origin > http://git-wip-us.apache.org/repos/asf/cassandra.git), and checked out to > tags/cassandra-3.0.14 - head is on f3e38cb638113c2a23855a104d6082da5bc10ddb. > > Then I have cloned the cassandra-dtest repo (origin git:// > github.com/riptano/cassandra-dtest.git). Head is on > 6843d76d0a85ad82edf889e8280b87786dc48486. > > I setup dtests according to this instructions: > https://github.com/riptano/cassandra-dtest/blob/master/INSTALL.md > > In addition, I have setup JAVA8_HOME and JAVA_HOME variables to public jre > of my 1.8.0_144 jdk. > > I start testing using this command: > JAVA8_HOME=$JAVA8_HOME nosetests --with-flaky --with-xunit > --xunit-file=out.xunit.xml --force-flaky --max-runs=3 --verbose > --debug-log=err.debug.nose.txt 1> out.txt 2> err.txt > I run the tests on x86_64 CentOS 7 with 7.6G of RAM. > > Problem symptoms: > During the "normal" run (I have got only 1 "normal" run in 5 attempts), > err.txt is updated constantly with name of the test recently completed, and > in the end out.xunit.xml file appears, with test summary results. nosetests > process exits. > > During the "problem" run tests stop progressing (err.txt was not modified > for 10 hours), out.xunit.xml is not appearing, nosetests process runs. I > killed java processes, but nothing changed for 2 hours - nosetests process > still runs, but files are unchanged. > > Any help would be appreciated, > Sergey > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
