I'll give it a try. Isn't it somewhat concerning that this would heal to missing gfids and failing self-heals though?
On 19.02.2013, at 21:22, Anand Avati <[email protected]> wrote: > We run dbench in the pre-commit script for every patch. Looks like you are > running dbench on the same directory from multiple clients. Is that even > supposed to work? Did you instead lunch to run each dbench within their own > subdirectory? > > Avati > > On Tue, Feb 19, 2013 at 12:07 PM, Marc Seeger <[email protected]> wrote: > To test gluster's behavior under heavy load, I'm currently doing this on two > machines sharing a common /mnt/gfs gluster mount: > > ssh bal-6.example.com apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs > ssh bal-7.example.com apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs > > > One of the processes usually dies pretty quickly like this: > > [608] open /mnt/gfs/clients/client5/~dmtmp/PWRPNT/PCBENCHM.PPT failed for > handle 10003 (No such file or directory) > (610) ERROR: handle 10003 was not found, > Child failed with status 1 > > > And the logs are full of things like this (ignore the initial timestamp, > that's from our logging): > > [2013-02-19 14:38:38.714493] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background data missing-entry gfid self-heal failed on > /clients/client5/~dmtmp/PM/MOVED.DOC, > [2013-02-19 14:38:38.724494] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client3/~dmtmp, > [2013-02-19 14:38:38.734495] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background data missing-entry gfid self-heal failed on > /clients/client4/~dmtmp/PM/EVENTS.DOC, > [2013-02-19 14:38:38.734495] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background data missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/PM/MOVED.DOC, > [2013-02-19 14:38:38.734495] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background data missing-entry gfid self-heal failed on > /clients/client1/~dmtmp/PM/MOVED.DOC, > [2013-02-19 14:38:38.734495] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background data missing-entry gfid self-heal failed on > /clients/client0/~dmtmp/PM/MOVED.DOC, > [2013-02-19 14:38:38.734495] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client4/~dmtmp/PM, [build-2 > system.rb:340], I, > [2013-02-19T14:39:50.189970 #20802] INFO -- : > [2013-02-19 14:38:36.041890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /, > [2013-02-19 14:38:36.041890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /, > [2013-02-19 14:38:36.041890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /, > [2013-02-19 14:38:36.041890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /, > [2013-02-19 14:38:36.041890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /, > [2013-02-19 14:38:36.051890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients, > [2013-02-19 14:38:36.071890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2, > [2013-02-19 14:38:36.071890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3, > [2013-02-19 14:38:36.071890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client2, > [2013-02-19 14:38:36.081890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client3, > [2013-02-19 14:38:36.091890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp, > [2013-02-19 14:38:36.091890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp, > [2013-02-19 14:38:36.101890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client2/~dmtmp, > [2013-02-19 14:38:36.101890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client3/~dmtmp, > [2013-02-19 14:38:36.111890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/WORD, > [2013-02-19 14:38:36.111890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/WORD, > [2013-02-19 14:38:36.131890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client2/~dmtmp/WORD, > [2013-02-19 14:38:36.141890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client3/~dmtmp/WORD, > [2013-02-19 14:38:36.151890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/WORD/CHAP10.DOC, > [2013-02-19 14:38:36.151890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/WORD/CHAP10.DOC, > [2013-02-19 14:38:36.161890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/WORD/BASEMACH.DOC, > [2013-02-19 14:38:36.161890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/WORD/BASEMACH.DOC, > [2013-02-19 14:38:36.171890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entr [build-2 system.rb:340], I, > [2013-02-19T14:39:50.189970 #20802] INFO -- : y missing-entry gfid self-heal > failed on /clients/client2/~dmtmp/WORD/FACTS.DOC, > [2013-02-19 14:38:36.181890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/WORD/FACTS.DOC, > [2013-02-19 14:38:36.201890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/EXCEL, > [2013-02-19 14:38:36.201890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/EXCEL, > [2013-02-19 14:38:36.201890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client2/~dmtmp/EXCEL, > [2013-02-19 14:38:36.201890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client3/~dmtmp/EXCEL, > [2013-02-19 14:38:36.211890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client0/~dmtmp, > [2013-02-19 14:38:36.211890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/EXCEL/PCMAGCD.XLS, > [2013-02-19 14:38:36.211890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/EXCEL/PCMAGCD.XLS, > [2013-02-19 14:38:36.241890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/EXCEL/SALES.XLS, > [2013-02-19 14:38:36.241890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/EXCEL/SALES.XLS, > [2013-02-19 14:38:36.271890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/PWRPNT, > [2013-02-19 14:38:36.271890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/PWRPNT, > [2013-02-19 14:38:36.281890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client2/~dmtmp/PWRPNT, > [2013-02-19 14:38:36.281890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background entry self-heal failed on /clients/client3/~dmtmp/PWRPNT, > [2013-02-19 14:38:36.291890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/PWRPNT/PCBENCHM.PPT, > [2013-02-19 14:38:36.311890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/PWRPNT/PCBENCHM.PPT, > [2013-02-19 14:38:36.351890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/PWRPNT/ZD16.BMP, > [2013-02-19 14:38:36.351890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client3/~dmtmp/PWRPNT/ZD16.BMP, > [2013-02-19 14:38:36.381890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid self-heal failed on > /clients/client2/~dmtmp/PWRPNT/PPTOOLS1.PPA, > [2013-02-19 14:38:36.391890] E > [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: > background meta-data data entry missing-entry gfid sel [build-2 > system.rb:340] > > > > Any ideas? Can somebody confirm this happens for them too? > > The setup is ubuntu lucid machines running 3.3.1 from this PPA: > https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3 > > > _______________________________________________ > Gluster-users mailing list > [email protected] > http://supercolony.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
