Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
On 4 Mar 2015, at 15:25, Shyam srang...@redhat.com wrote: On 03/03/2015 11:27 PM, Justin Clift wrote: 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 FWIW, this is the same as https://bugzilla.redhat.com/show_bug.cgi?id=1195415 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 So is this one. i.e same as https://bugzilla.redhat.com/show_bug.cgi?id=1195415 Thanks Shyam. Somehow missed your email earlier, but all good now. :) + Justin -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
On 03/04/2015 09:57 AM, Justin Clift wrote: Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 Nice catch by regression. Fix: http://review.gluster.org/9817 Pranith * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
On 03/04/2015 10:25 AM, Nithya Balachandran wrote: Fix for the spurious bug-1117851.t failure at http://review.gluster.org/#/c/9798/ Thanks, Shyam and Nithya for a quick turnaround on this spurious failure! -Vijay ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Fix for the spurious bug-1117851.t failure at http://review.gluster.org/#/c/9798/ Regards, Nithya - Original Message - From: Justin Clift jus...@gluster.org To: Nithya Balachandran nbala...@redhat.com Cc: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, 4 March, 2015 10:12:17 AM Subject: Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Thanks. :) If you need a VM setup in Rackspace for you to investigate on, it's easy to do. Let me know if so. :) + Justin On 4 Mar 2015, at 04:37, Nithya Balachandran nbala...@redhat.com wrote: I'll take a look at tests/bugs/distribute/bug-1117851.t Regards, Nithya - Original Message - From: Justin Clift jus...@gluster.org To: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, 4 March, 2015 9:57:00 AM Subject: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
On 03/03/2015 11:27 PM, Justin Clift wrote: 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 FWIW, this is the same as https://bugzilla.redhat.com/show_bug.cgi?id=1195415 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 So is this one. i.e same as https://bugzilla.redhat.com/show_bug.cgi?id=1195415 Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Hi, I had a look at tests/bugs/distribute/bug-1117851.t The test fails at : EXPECT_WITHIN 75 done cat $M0/status_0 The test uses a status file to check if the file rename operation (where a 1000 files are renamed) which runs in the background is over. The status file $M0/status_0 is created before the rename begins and the string running is written to it. Once the rename is done, the string done is written to the file. So it turns out the renames are actually finishing well in time - roughly 40 seconds. But the status_0 file is not present so cat fails on the file. The logs for two regression runs that failed confirm this (http://build.gluster.org/job/rackspace-regression-2GB/951/console and http://build.gluster.org/job/rackspace-regression-2GB/983/console). cat: /mnt/glusterfs/0/status_0: No such file or directory [14:53:50] ./tests/bugs/distribute/bug-1117851.t . not ok 15 Got instead of done Failed 1/24 subtests The test runs successfully on my local setup and has failed only twice on the VM Justin provided(out of about 50 runs), so I am still looking into why it cannot find the file. Regards, Nithya - Original Message - From: Justin Clift jus...@gluster.org To: Nithya Balachandran nbala...@redhat.com Cc: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, 4 March, 2015 10:12:17 AM Subject: Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Thanks. :) If you need a VM setup in Rackspace for you to investigate on, it's easy to do. Let me know if so. :) + Justin On 4 Mar 2015, at 04:37, Nithya Balachandran nbala...@redhat.com wrote: I'll take a look at tests/bugs/distribute/bug-1117851.t Regards, Nithya - Original Message - From: Justin Clift jus...@gluster.org To: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, 4 March, 2015 9:57:00 AM Subject: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t
[Gluster-devel] Spurious failure report for master branch - 2015-03-03
Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Thanks. :) If you need a VM setup in Rackspace for you to investigate on, it's easy to do. Let me know if so. :) + Justin On 4 Mar 2015, at 04:37, Nithya Balachandran nbala...@redhat.com wrote: I'll take a look at tests/bugs/distribute/bug-1117851.t Regards, Nithya - Original Message - From: Justin Clift jus...@gluster.org To: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, 4 March, 2015 9:57:00 AM Subject: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Few more test cases causing spurious failures: ./tests/basic/ec/ec-5-1.t Failed test: 69 ./tests/basic/ec/ec-5-2.t Failed test: 69 ./tests/bugs/disperse/bug-1187474.t Failed tests: 11-12 ./tests/basic/ec/nfs.t Failed test: 9 The above failures were seen for the patches which were ineffective, i.e. the code that was modified was never executed as it had no callers. Regards, Poornima - Original Message - From: Justin Clift jus...@gluster.org To: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, March 4, 2015 9:57:00 AM Subject: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
I'll take a look at tests/bugs/distribute/bug-1117851.t Regards, Nithya - Original Message - From: Justin Clift jus...@gluster.org To: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, 4 March, 2015 9:57:00 AM Subject: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Looks like I will need a VM - the test passes consistently on my local setup. Can you please send me the details? Regards, Nithya - Original Message - From: Justin Clift jus...@gluster.org To: Nithya Balachandran nbala...@redhat.com Cc: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, 4 March, 2015 10:12:17 AM Subject: Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Thanks. :) If you need a VM setup in Rackspace for you to investigate on, it's easy to do. Let me know if so. :) + Justin On 4 Mar 2015, at 04:37, Nithya Balachandran nbala...@redhat.com wrote: I'll take a look at tests/bugs/distribute/bug-1117851.t Regards, Nithya - Original Message - From: Justin Clift jus...@gluster.org To: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, 4 March, 2015 9:57:00 AM Subject: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org
